Keras and the Last Number Problem

Keras and the Last Number Problem#

Let’s see if we can do better than our simple hidden layer NN with the last number problem.

import numpy as np
import keras
from keras.utils import to_categorical

We’ll use the same data class

class ModelDataCategorical:
    """this is the model data for our "last number" training set.  We
    produce input of length N, consisting of numbers 0-9 and store
    the result in a 10-element array as categorical data.

    """
    def __init__(self, N=10):
        self.N = N
        
        # our model input data
        self.x = np.random.randint(0, high=10, size=N)
        self.x_scaled = self.x / 10 + 0.05
        
        # our scaled model output data
        self.y = np.array([self.x[-1]])
        self.y_scaled = np.zeros(10) + 0.01
        self.y_scaled[self.x[-1]] = 0.99
        
    def interpret_result(self, out):
        """take the network output and return the number we predict"""
        return np.argmax(out)

For Keras, we need to pack the scaled data (both input and output) into arrays. We’ll use the Keras to_categorical() to make the data categorical.

Let’s make both a training set and a test set

x_train = []
y_train = []
for _ in range(10000):
    m = ModelDataCategorical()
    x_train.append(m.x_scaled)
    y_train.append(m.y)

x_train = np.asarray(x_train)
y_train = to_categorical(y_train, 10)
x_test = []
y_test = []
for _ in range(1000):
    m = ModelDataCategorical()
    x_test.append(m.x_scaled)
    y_test.append(m.y)

x_test = np.asarray(x_test)
y_test = to_categorical(y_test, 10)

Check to make sure the data looks like we expect:

x_train[0]
array([0.15, 0.45, 0.95, 0.75, 0.15, 0.85, 0.45, 0.25, 0.45, 0.45])
y_train[0]
array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])

Creating the network#

Now let’s build our network. We’ll use just a single hidden layer, but instead of the sigmoid used before, we’ll use RELU and the softmax activations.

from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation
from keras.optimizers import RMSprop
model = Sequential()
model.add(Input((10,)))
model.add(Dense(100, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(10, activation="softmax"))
rms = RMSprop()
model.compile(loss='categorical_crossentropy',
              optimizer=rms, metrics=['accuracy'])
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 100)            │         1,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 100)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 10)             │         1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 2,110 (8.24 KB)
 Trainable params: 2,110 (8.24 KB)
 Non-trainable params: 0 (0.00 B)

Now we have ~ 2k parameters to fit.

Training#

Now we can train and test each epoch to see how we do

epochs = 100
batch_size = 256
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
          validation_data=(x_test, y_test), verbose=2)
Epoch 1/100
40/40 - 0s - 4ms/step - accuracy: 0.1671 - loss: 2.2487 - val_accuracy: 0.2460 - val_loss: 2.1895
Epoch 2/100
40/40 - 0s - 4ms/step - accuracy: 0.2485 - loss: 2.1463 - val_accuracy: 0.2840 - val_loss: 2.0883
Epoch 3/100
40/40 - 0s - 4ms/step - accuracy: 0.2727 - loss: 2.0516 - val_accuracy: 0.2930 - val_loss: 1.9879
Epoch 4/100
40/40 - 0s - 4ms/step - accuracy: 0.2893 - loss: 1.9562 - val_accuracy: 0.3070 - val_loss: 1.8951
Epoch 5/100
40/40 - 0s - 3ms/step - accuracy: 0.3211 - loss: 1.8642 - val_accuracy: 0.3650 - val_loss: 1.8070
Epoch 6/100
40/40 - 0s - 3ms/step - accuracy: 0.3478 - loss: 1.7784 - val_accuracy: 0.3730 - val_loss: 1.7213
Epoch 7/100
40/40 - 0s - 3ms/step - accuracy: 0.3739 - loss: 1.6991 - val_accuracy: 0.4100 - val_loss: 1.6423
Epoch 8/100
40/40 - 0s - 3ms/step - accuracy: 0.4026 - loss: 1.6318 - val_accuracy: 0.4220 - val_loss: 1.5774
Epoch 9/100
40/40 - 0s - 3ms/step - accuracy: 0.4271 - loss: 1.5662 - val_accuracy: 0.4640 - val_loss: 1.5134
Epoch 10/100
40/40 - 0s - 3ms/step - accuracy: 0.4499 - loss: 1.5090 - val_accuracy: 0.5190 - val_loss: 1.4621
Epoch 11/100
40/40 - 0s - 3ms/step - accuracy: 0.4741 - loss: 1.4557 - val_accuracy: 0.5630 - val_loss: 1.4090
Epoch 12/100
40/40 - 0s - 3ms/step - accuracy: 0.4935 - loss: 1.4036 - val_accuracy: 0.4880 - val_loss: 1.3543
Epoch 13/100
40/40 - 0s - 3ms/step - accuracy: 0.5086 - loss: 1.3554 - val_accuracy: 0.5600 - val_loss: 1.3109
Epoch 14/100
40/40 - 0s - 4ms/step - accuracy: 0.5405 - loss: 1.3097 - val_accuracy: 0.5430 - val_loss: 1.2716
Epoch 15/100
40/40 - 0s - 3ms/step - accuracy: 0.5624 - loss: 1.2666 - val_accuracy: 0.6930 - val_loss: 1.2243
Epoch 16/100
40/40 - 0s - 3ms/step - accuracy: 0.5808 - loss: 1.2288 - val_accuracy: 0.6390 - val_loss: 1.1840
Epoch 17/100
40/40 - 0s - 4ms/step - accuracy: 0.5951 - loss: 1.1916 - val_accuracy: 0.5920 - val_loss: 1.1491
Epoch 18/100
40/40 - 0s - 4ms/step - accuracy: 0.6049 - loss: 1.1606 - val_accuracy: 0.7180 - val_loss: 1.1279
Epoch 19/100
40/40 - 0s - 4ms/step - accuracy: 0.6350 - loss: 1.1261 - val_accuracy: 0.7470 - val_loss: 1.0817
Epoch 20/100
40/40 - 0s - 4ms/step - accuracy: 0.6544 - loss: 1.0947 - val_accuracy: 0.6850 - val_loss: 1.0562
Epoch 21/100
40/40 - 0s - 4ms/step - accuracy: 0.6745 - loss: 1.0652 - val_accuracy: 0.7790 - val_loss: 1.0199
Epoch 22/100
40/40 - 0s - 3ms/step - accuracy: 0.6840 - loss: 1.0369 - val_accuracy: 0.7370 - val_loss: 1.0046
Epoch 23/100
40/40 - 0s - 4ms/step - accuracy: 0.6989 - loss: 1.0118 - val_accuracy: 0.8210 - val_loss: 0.9711
Epoch 24/100
40/40 - 0s - 4ms/step - accuracy: 0.7194 - loss: 0.9833 - val_accuracy: 0.7940 - val_loss: 0.9464
Epoch 25/100
40/40 - 0s - 4ms/step - accuracy: 0.7313 - loss: 0.9587 - val_accuracy: 0.8300 - val_loss: 0.9243
Epoch 26/100
40/40 - 0s - 4ms/step - accuracy: 0.7480 - loss: 0.9335 - val_accuracy: 0.8620 - val_loss: 0.8927
Epoch 27/100
40/40 - 0s - 4ms/step - accuracy: 0.7623 - loss: 0.9095 - val_accuracy: 0.8420 - val_loss: 0.8779
Epoch 28/100
40/40 - 0s - 3ms/step - accuracy: 0.7791 - loss: 0.8879 - val_accuracy: 0.8740 - val_loss: 0.8538
Epoch 29/100
40/40 - 0s - 3ms/step - accuracy: 0.7906 - loss: 0.8653 - val_accuracy: 0.8860 - val_loss: 0.8274
Epoch 30/100
40/40 - 0s - 3ms/step - accuracy: 0.8033 - loss: 0.8442 - val_accuracy: 0.8730 - val_loss: 0.8054
Epoch 31/100
40/40 - 0s - 3ms/step - accuracy: 0.8192 - loss: 0.8214 - val_accuracy: 0.8780 - val_loss: 0.7818
Epoch 32/100
40/40 - 0s - 3ms/step - accuracy: 0.8289 - loss: 0.7989 - val_accuracy: 0.8790 - val_loss: 0.7638
Epoch 33/100
40/40 - 0s - 4ms/step - accuracy: 0.8500 - loss: 0.7785 - val_accuracy: 0.9420 - val_loss: 0.7426
Epoch 34/100
40/40 - 0s - 4ms/step - accuracy: 0.8651 - loss: 0.7561 - val_accuracy: 0.8890 - val_loss: 0.7371
Epoch 35/100
40/40 - 0s - 4ms/step - accuracy: 0.8713 - loss: 0.7390 - val_accuracy: 0.9360 - val_loss: 0.7087
Epoch 36/100
40/40 - 0s - 3ms/step - accuracy: 0.8822 - loss: 0.7213 - val_accuracy: 0.9570 - val_loss: 0.6852
Epoch 37/100
40/40 - 0s - 3ms/step - accuracy: 0.9062 - loss: 0.6939 - val_accuracy: 0.9730 - val_loss: 0.6553
Epoch 38/100
40/40 - 0s - 3ms/step - accuracy: 0.9128 - loss: 0.6767 - val_accuracy: 0.9640 - val_loss: 0.6392
Epoch 39/100
40/40 - 0s - 3ms/step - accuracy: 0.9172 - loss: 0.6592 - val_accuracy: 0.9740 - val_loss: 0.6251
Epoch 40/100
40/40 - 0s - 3ms/step - accuracy: 0.9307 - loss: 0.6398 - val_accuracy: 0.9850 - val_loss: 0.5988
Epoch 41/100
40/40 - 0s - 4ms/step - accuracy: 0.9358 - loss: 0.6213 - val_accuracy: 0.9820 - val_loss: 0.5922
Epoch 42/100
40/40 - 0s - 4ms/step - accuracy: 0.9470 - loss: 0.6040 - val_accuracy: 0.9870 - val_loss: 0.5683
Epoch 43/100
40/40 - 0s - 4ms/step - accuracy: 0.9541 - loss: 0.5860 - val_accuracy: 0.9850 - val_loss: 0.5584
Epoch 44/100
40/40 - 0s - 3ms/step - accuracy: 0.9573 - loss: 0.5696 - val_accuracy: 0.9920 - val_loss: 0.5386
Epoch 45/100
40/40 - 0s - 3ms/step - accuracy: 0.9631 - loss: 0.5552 - val_accuracy: 0.9900 - val_loss: 0.5271
Epoch 46/100
40/40 - 0s - 4ms/step - accuracy: 0.9692 - loss: 0.5374 - val_accuracy: 0.9990 - val_loss: 0.5017
Epoch 47/100
40/40 - 0s - 4ms/step - accuracy: 0.9708 - loss: 0.5220 - val_accuracy: 0.9990 - val_loss: 0.4970
Epoch 48/100
40/40 - 0s - 4ms/step - accuracy: 0.9753 - loss: 0.5071 - val_accuracy: 0.9980 - val_loss: 0.4712
Epoch 49/100
40/40 - 0s - 4ms/step - accuracy: 0.9780 - loss: 0.4919 - val_accuracy: 0.9980 - val_loss: 0.4648
Epoch 50/100
40/40 - 0s - 4ms/step - accuracy: 0.9825 - loss: 0.4757 - val_accuracy: 0.9960 - val_loss: 0.4531
Epoch 51/100
40/40 - 0s - 4ms/step - accuracy: 0.9849 - loss: 0.4626 - val_accuracy: 1.0000 - val_loss: 0.4342
Epoch 52/100
40/40 - 0s - 4ms/step - accuracy: 0.9843 - loss: 0.4482 - val_accuracy: 1.0000 - val_loss: 0.4150
Epoch 53/100
40/40 - 0s - 4ms/step - accuracy: 0.9854 - loss: 0.4337 - val_accuracy: 0.9990 - val_loss: 0.4054
Epoch 54/100
40/40 - 0s - 4ms/step - accuracy: 0.9888 - loss: 0.4196 - val_accuracy: 0.9960 - val_loss: 0.3935
Epoch 55/100
40/40 - 0s - 4ms/step - accuracy: 0.9883 - loss: 0.4061 - val_accuracy: 0.9980 - val_loss: 0.3814
Epoch 56/100
40/40 - 0s - 4ms/step - accuracy: 0.9884 - loss: 0.3946 - val_accuracy: 1.0000 - val_loss: 0.3648
Epoch 57/100
40/40 - 0s - 4ms/step - accuracy: 0.9916 - loss: 0.3803 - val_accuracy: 1.0000 - val_loss: 0.3650
Epoch 58/100
40/40 - 0s - 4ms/step - accuracy: 0.9937 - loss: 0.3680 - val_accuracy: 0.9970 - val_loss: 0.3489
Epoch 59/100
40/40 - 0s - 4ms/step - accuracy: 0.9925 - loss: 0.3568 - val_accuracy: 1.0000 - val_loss: 0.3318
Epoch 60/100
40/40 - 0s - 4ms/step - accuracy: 0.9946 - loss: 0.3418 - val_accuracy: 1.0000 - val_loss: 0.3154
Epoch 61/100
40/40 - 0s - 4ms/step - accuracy: 0.9951 - loss: 0.3326 - val_accuracy: 1.0000 - val_loss: 0.3037
Epoch 62/100
40/40 - 0s - 4ms/step - accuracy: 0.9935 - loss: 0.3222 - val_accuracy: 1.0000 - val_loss: 0.2945
Epoch 63/100
40/40 - 0s - 4ms/step - accuracy: 0.9950 - loss: 0.3103 - val_accuracy: 1.0000 - val_loss: 0.2826
Epoch 64/100
40/40 - 0s - 4ms/step - accuracy: 0.9951 - loss: 0.3005 - val_accuracy: 1.0000 - val_loss: 0.2702
Epoch 65/100
40/40 - 0s - 4ms/step - accuracy: 0.9944 - loss: 0.2898 - val_accuracy: 1.0000 - val_loss: 0.2643
Epoch 66/100
40/40 - 0s - 4ms/step - accuracy: 0.9967 - loss: 0.2785 - val_accuracy: 1.0000 - val_loss: 0.2469
Epoch 67/100
40/40 - 0s - 4ms/step - accuracy: 0.9968 - loss: 0.2690 - val_accuracy: 1.0000 - val_loss: 0.2404
Epoch 68/100
40/40 - 0s - 4ms/step - accuracy: 0.9962 - loss: 0.2609 - val_accuracy: 1.0000 - val_loss: 0.2305
Epoch 69/100
40/40 - 0s - 4ms/step - accuracy: 0.9961 - loss: 0.2515 - val_accuracy: 1.0000 - val_loss: 0.2258
Epoch 70/100
40/40 - 0s - 4ms/step - accuracy: 0.9979 - loss: 0.2419 - val_accuracy: 1.0000 - val_loss: 0.2171
Epoch 71/100
40/40 - 0s - 4ms/step - accuracy: 0.9969 - loss: 0.2349 - val_accuracy: 1.0000 - val_loss: 0.2056
Epoch 72/100
40/40 - 0s - 4ms/step - accuracy: 0.9979 - loss: 0.2249 - val_accuracy: 1.0000 - val_loss: 0.1998
Epoch 73/100
40/40 - 0s - 4ms/step - accuracy: 0.9973 - loss: 0.2186 - val_accuracy: 1.0000 - val_loss: 0.1935
Epoch 74/100
40/40 - 0s - 4ms/step - accuracy: 0.9971 - loss: 0.2094 - val_accuracy: 1.0000 - val_loss: 0.1880
Epoch 75/100
40/40 - 0s - 4ms/step - accuracy: 0.9982 - loss: 0.2006 - val_accuracy: 1.0000 - val_loss: 0.1759
Epoch 76/100
40/40 - 0s - 4ms/step - accuracy: 0.9981 - loss: 0.1941 - val_accuracy: 1.0000 - val_loss: 0.1738
Epoch 77/100
40/40 - 0s - 4ms/step - accuracy: 0.9979 - loss: 0.1876 - val_accuracy: 1.0000 - val_loss: 0.1685
Epoch 78/100
40/40 - 0s - 4ms/step - accuracy: 0.9989 - loss: 0.1800 - val_accuracy: 1.0000 - val_loss: 0.1531
Epoch 79/100
40/40 - 0s - 4ms/step - accuracy: 0.9984 - loss: 0.1740 - val_accuracy: 1.0000 - val_loss: 0.1560
Epoch 80/100
40/40 - 0s - 4ms/step - accuracy: 0.9987 - loss: 0.1671 - val_accuracy: 1.0000 - val_loss: 0.1437
Epoch 81/100
40/40 - 0s - 4ms/step - accuracy: 0.9987 - loss: 0.1607 - val_accuracy: 1.0000 - val_loss: 0.1360
Epoch 82/100
40/40 - 0s - 4ms/step - accuracy: 0.9982 - loss: 0.1549 - val_accuracy: 1.0000 - val_loss: 0.1339
Epoch 83/100
40/40 - 0s - 4ms/step - accuracy: 0.9991 - loss: 0.1487 - val_accuracy: 1.0000 - val_loss: 0.1284
Epoch 84/100
40/40 - 0s - 4ms/step - accuracy: 0.9987 - loss: 0.1437 - val_accuracy: 1.0000 - val_loss: 0.1186
Epoch 85/100
40/40 - 0s - 4ms/step - accuracy: 0.9989 - loss: 0.1368 - val_accuracy: 1.0000 - val_loss: 0.1180
Epoch 86/100
40/40 - 0s - 4ms/step - accuracy: 0.9992 - loss: 0.1325 - val_accuracy: 1.0000 - val_loss: 0.1117
Epoch 87/100
40/40 - 0s - 4ms/step - accuracy: 0.9986 - loss: 0.1290 - val_accuracy: 1.0000 - val_loss: 0.1061
Epoch 88/100
40/40 - 0s - 4ms/step - accuracy: 0.9991 - loss: 0.1222 - val_accuracy: 1.0000 - val_loss: 0.0982
Epoch 89/100
40/40 - 0s - 4ms/step - accuracy: 0.9993 - loss: 0.1173 - val_accuracy: 1.0000 - val_loss: 0.1010
Epoch 90/100
40/40 - 0s - 4ms/step - accuracy: 0.9995 - loss: 0.1138 - val_accuracy: 1.0000 - val_loss: 0.0916
Epoch 91/100
40/40 - 0s - 4ms/step - accuracy: 0.9994 - loss: 0.1102 - val_accuracy: 1.0000 - val_loss: 0.0865
Epoch 92/100
40/40 - 0s - 4ms/step - accuracy: 0.9999 - loss: 0.1051 - val_accuracy: 1.0000 - val_loss: 0.0819
Epoch 93/100
40/40 - 0s - 4ms/step - accuracy: 0.9995 - loss: 0.1008 - val_accuracy: 1.0000 - val_loss: 0.0785
Epoch 94/100
40/40 - 0s - 4ms/step - accuracy: 0.9995 - loss: 0.0973 - val_accuracy: 1.0000 - val_loss: 0.0795
Epoch 95/100
40/40 - 0s - 4ms/step - accuracy: 0.9988 - loss: 0.0941 - val_accuracy: 1.0000 - val_loss: 0.0750
Epoch 96/100
40/40 - 0s - 4ms/step - accuracy: 0.9993 - loss: 0.0902 - val_accuracy: 1.0000 - val_loss: 0.0691
Epoch 97/100
40/40 - 0s - 4ms/step - accuracy: 0.9995 - loss: 0.0856 - val_accuracy: 1.0000 - val_loss: 0.0657
Epoch 98/100
40/40 - 0s - 4ms/step - accuracy: 0.9996 - loss: 0.0823 - val_accuracy: 1.0000 - val_loss: 0.0631
Epoch 99/100
40/40 - 0s - 4ms/step - accuracy: 0.9996 - loss: 0.0795 - val_accuracy: 1.0000 - val_loss: 0.0615
Epoch 100/100
40/40 - 0s - 4ms/step - accuracy: 0.9996 - loss: 0.0768 - val_accuracy: 1.0000 - val_loss: 0.0623
<keras.src.callbacks.history.History at 0x7f66743016a0>

As we see, the network is essentially perfect now.