Keras and the Last Number Problem#
Let’s see if we can do better than our simple hidden layer NN with the last number problem.
import numpy as np
import keras
from keras.utils import to_categorical
We’ll use the same data class
class ModelDataCategorical:
"""this is the model data for our "last number" training set. We
produce input of length N, consisting of numbers 0-9 and store
the result in a 10-element array as categorical data.
"""
def __init__(self, N=10):
self.N = N
# our model input data
self.x = np.random.randint(0, high=10, size=N)
self.x_scaled = self.x / 10 + 0.05
# our scaled model output data
self.y = np.array([self.x[-1]])
self.y_scaled = np.zeros(10) + 0.01
self.y_scaled[self.x[-1]] = 0.99
def interpret_result(self, out):
"""take the network output and return the number we predict"""
return np.argmax(out)
For Keras, we need to pack the scaled data (both input and output) into arrays. We’ll use
the Keras to_categorical() to make the data categorical.
Let’s make both a training set and a test set
x_train = []
y_train = []
for _ in range(10000):
m = ModelDataCategorical()
x_train.append(m.x_scaled)
y_train.append(m.y)
x_train = np.asarray(x_train)
y_train = to_categorical(y_train, 10)
x_test = []
y_test = []
for _ in range(1000):
m = ModelDataCategorical()
x_test.append(m.x_scaled)
y_test.append(m.y)
x_test = np.asarray(x_test)
y_test = to_categorical(y_test, 10)
Check to make sure the data looks like we expect:
x_train[0]
array([0.65, 0.45, 0.55, 0.05, 0.25, 0.25, 0.05, 0.65, 0.55, 0.35])
y_train[0]
array([0., 0., 0., 1., 0., 0., 0., 0., 0., 0.])
Creating the network#
Now let’s build our network. We’ll use just a single hidden layer, but instead of the sigmoid used before, we’ll use RELU and the softmax activations.
from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation
from keras.optimizers import RMSprop
model = Sequential()
model.add(Input((10,)))
model.add(Dense(100, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(10, activation="softmax"))
rms = RMSprop()
model.compile(loss='categorical_crossentropy',
optimizer=rms, metrics=['accuracy'])
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 100) │ 1,100 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 100) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 10) │ 1,010 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,110 (8.24 KB)
Trainable params: 2,110 (8.24 KB)
Non-trainable params: 0 (0.00 B)
Now we have ~ 2k parameters to fit.
Training#
Now we can train and test each epoch to see how we do
epochs = 100
batch_size = 256
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
validation_data=(x_test, y_test), verbose=2)
Epoch 1/100
40/40 - 0s - 6ms/step - accuracy: 0.1703 - loss: 2.2582 - val_accuracy: 0.2480 - val_loss: 2.2033
Epoch 2/100
40/40 - 0s - 6ms/step - accuracy: 0.2412 - loss: 2.1639 - val_accuracy: 0.2480 - val_loss: 2.1092
Epoch 3/100
40/40 - 0s - 6ms/step - accuracy: 0.2609 - loss: 2.0643 - val_accuracy: 0.2490 - val_loss: 2.0093
Epoch 4/100
40/40 - 0s - 6ms/step - accuracy: 0.2757 - loss: 1.9646 - val_accuracy: 0.3290 - val_loss: 1.9105
Epoch 5/100
40/40 - 0s - 6ms/step - accuracy: 0.3029 - loss: 1.8743 - val_accuracy: 0.3190 - val_loss: 1.8225
Epoch 6/100
40/40 - 0s - 6ms/step - accuracy: 0.3281 - loss: 1.7831 - val_accuracy: 0.3480 - val_loss: 1.7366
Epoch 7/100
40/40 - 0s - 6ms/step - accuracy: 0.3577 - loss: 1.7073 - val_accuracy: 0.3860 - val_loss: 1.6653
Epoch 8/100
40/40 - 0s - 6ms/step - accuracy: 0.3935 - loss: 1.6344 - val_accuracy: 0.4550 - val_loss: 1.5941
Epoch 9/100
40/40 - 0s - 6ms/step - accuracy: 0.4204 - loss: 1.5697 - val_accuracy: 0.4490 - val_loss: 1.5334
Epoch 10/100
40/40 - 0s - 6ms/step - accuracy: 0.4428 - loss: 1.5122 - val_accuracy: 0.5180 - val_loss: 1.4759
Epoch 11/100
40/40 - 0s - 6ms/step - accuracy: 0.4707 - loss: 1.4558 - val_accuracy: 0.5420 - val_loss: 1.4236
Epoch 12/100
40/40 - 0s - 6ms/step - accuracy: 0.4899 - loss: 1.4051 - val_accuracy: 0.4980 - val_loss: 1.3772
Epoch 13/100
40/40 - 0s - 6ms/step - accuracy: 0.5049 - loss: 1.3594 - val_accuracy: 0.5970 - val_loss: 1.3306
Epoch 14/100
40/40 - 0s - 6ms/step - accuracy: 0.5401 - loss: 1.3112 - val_accuracy: 0.5040 - val_loss: 1.2878
Epoch 15/100
40/40 - 0s - 6ms/step - accuracy: 0.5463 - loss: 1.2727 - val_accuracy: 0.6430 - val_loss: 1.2555
Epoch 16/100
40/40 - 0s - 6ms/step - accuracy: 0.5690 - loss: 1.2344 - val_accuracy: 0.6180 - val_loss: 1.2129
Epoch 17/100
40/40 - 0s - 6ms/step - accuracy: 0.5920 - loss: 1.1982 - val_accuracy: 0.6520 - val_loss: 1.1744
Epoch 18/100
40/40 - 0s - 6ms/step - accuracy: 0.6140 - loss: 1.1610 - val_accuracy: 0.6590 - val_loss: 1.1415
Epoch 19/100
40/40 - 0s - 6ms/step - accuracy: 0.6247 - loss: 1.1286 - val_accuracy: 0.6550 - val_loss: 1.1120
Epoch 20/100
40/40 - 0s - 6ms/step - accuracy: 0.6489 - loss: 1.0990 - val_accuracy: 0.7260 - val_loss: 1.0792
Epoch 21/100
40/40 - 0s - 6ms/step - accuracy: 0.6768 - loss: 1.0685 - val_accuracy: 0.7050 - val_loss: 1.0522
Epoch 22/100
40/40 - 0s - 6ms/step - accuracy: 0.6840 - loss: 1.0388 - val_accuracy: 0.8030 - val_loss: 1.0218
Epoch 23/100
40/40 - 0s - 6ms/step - accuracy: 0.7012 - loss: 1.0110 - val_accuracy: 0.7640 - val_loss: 0.9949
Epoch 24/100
40/40 - 0s - 6ms/step - accuracy: 0.7220 - loss: 0.9848 - val_accuracy: 0.7700 - val_loss: 0.9673
Epoch 25/100
40/40 - 0s - 6ms/step - accuracy: 0.7374 - loss: 0.9588 - val_accuracy: 0.8260 - val_loss: 0.9435
Epoch 26/100
40/40 - 0s - 6ms/step - accuracy: 0.7540 - loss: 0.9357 - val_accuracy: 0.8260 - val_loss: 0.9200
Epoch 27/100
40/40 - 0s - 6ms/step - accuracy: 0.7656 - loss: 0.9103 - val_accuracy: 0.8710 - val_loss: 0.8902
Epoch 28/100
40/40 - 0s - 6ms/step - accuracy: 0.7810 - loss: 0.8850 - val_accuracy: 0.8410 - val_loss: 0.8702
Epoch 29/100
40/40 - 0s - 6ms/step - accuracy: 0.7964 - loss: 0.8643 - val_accuracy: 0.8280 - val_loss: 0.8508
Epoch 30/100
40/40 - 0s - 6ms/step - accuracy: 0.8096 - loss: 0.8400 - val_accuracy: 0.8540 - val_loss: 0.8317
Epoch 31/100
40/40 - 0s - 6ms/step - accuracy: 0.8219 - loss: 0.8211 - val_accuracy: 0.8540 - val_loss: 0.8155
Epoch 32/100
40/40 - 0s - 6ms/step - accuracy: 0.8450 - loss: 0.7966 - val_accuracy: 0.9150 - val_loss: 0.7815
Epoch 33/100
40/40 - 0s - 6ms/step - accuracy: 0.8515 - loss: 0.7777 - val_accuracy: 0.8950 - val_loss: 0.7633
Epoch 34/100
40/40 - 0s - 6ms/step - accuracy: 0.8581 - loss: 0.7572 - val_accuracy: 0.8990 - val_loss: 0.7426
Epoch 35/100
40/40 - 0s - 6ms/step - accuracy: 0.8683 - loss: 0.7360 - val_accuracy: 0.9300 - val_loss: 0.7236
Epoch 36/100
40/40 - 0s - 6ms/step - accuracy: 0.8839 - loss: 0.7161 - val_accuracy: 0.9400 - val_loss: 0.7013
Epoch 37/100
40/40 - 0s - 6ms/step - accuracy: 0.8954 - loss: 0.6990 - val_accuracy: 0.9270 - val_loss: 0.6828
Epoch 38/100
40/40 - 0s - 6ms/step - accuracy: 0.8970 - loss: 0.6794 - val_accuracy: 0.9420 - val_loss: 0.6621
Epoch 39/100
40/40 - 0s - 6ms/step - accuracy: 0.9123 - loss: 0.6594 - val_accuracy: 0.9580 - val_loss: 0.6427
Epoch 40/100
40/40 - 0s - 6ms/step - accuracy: 0.9207 - loss: 0.6424 - val_accuracy: 0.9710 - val_loss: 0.6238
Epoch 41/100
40/40 - 0s - 6ms/step - accuracy: 0.9295 - loss: 0.6241 - val_accuracy: 0.9700 - val_loss: 0.6121
Epoch 42/100
40/40 - 0s - 6ms/step - accuracy: 0.9301 - loss: 0.6071 - val_accuracy: 0.9560 - val_loss: 0.5983
Epoch 43/100
40/40 - 0s - 6ms/step - accuracy: 0.9453 - loss: 0.5899 - val_accuracy: 0.9770 - val_loss: 0.5750
Epoch 44/100
40/40 - 0s - 6ms/step - accuracy: 0.9481 - loss: 0.5745 - val_accuracy: 0.9620 - val_loss: 0.5639
Epoch 45/100
40/40 - 0s - 6ms/step - accuracy: 0.9502 - loss: 0.5612 - val_accuracy: 0.9890 - val_loss: 0.5393
Epoch 46/100
40/40 - 0s - 6ms/step - accuracy: 0.9605 - loss: 0.5432 - val_accuracy: 0.9830 - val_loss: 0.5243
Epoch 47/100
40/40 - 0s - 6ms/step - accuracy: 0.9623 - loss: 0.5283 - val_accuracy: 0.9740 - val_loss: 0.5158
Epoch 48/100
40/40 - 0s - 6ms/step - accuracy: 0.9646 - loss: 0.5130 - val_accuracy: 0.9970 - val_loss: 0.4906
Epoch 49/100
40/40 - 0s - 6ms/step - accuracy: 0.9714 - loss: 0.4981 - val_accuracy: 0.9900 - val_loss: 0.4773
Epoch 50/100
40/40 - 0s - 6ms/step - accuracy: 0.9723 - loss: 0.4837 - val_accuracy: 0.9960 - val_loss: 0.4718
Epoch 51/100
40/40 - 0s - 6ms/step - accuracy: 0.9725 - loss: 0.4691 - val_accuracy: 0.9990 - val_loss: 0.4473
Epoch 52/100
40/40 - 0s - 6ms/step - accuracy: 0.9837 - loss: 0.4551 - val_accuracy: 0.9970 - val_loss: 0.4403
Epoch 53/100
40/40 - 0s - 6ms/step - accuracy: 0.9819 - loss: 0.4414 - val_accuracy: 0.9930 - val_loss: 0.4287
Epoch 54/100
40/40 - 0s - 6ms/step - accuracy: 0.9806 - loss: 0.4294 - val_accuracy: 0.9990 - val_loss: 0.4105
Epoch 55/100
40/40 - 0s - 6ms/step - accuracy: 0.9847 - loss: 0.4163 - val_accuracy: 1.0000 - val_loss: 0.3919
Epoch 56/100
40/40 - 0s - 6ms/step - accuracy: 0.9865 - loss: 0.4040 - val_accuracy: 0.9970 - val_loss: 0.3942
Epoch 57/100
40/40 - 0s - 6ms/step - accuracy: 0.9870 - loss: 0.3933 - val_accuracy: 0.9990 - val_loss: 0.3688
Epoch 58/100
40/40 - 0s - 6ms/step - accuracy: 0.9893 - loss: 0.3788 - val_accuracy: 0.9990 - val_loss: 0.3599
Epoch 59/100
40/40 - 0s - 6ms/step - accuracy: 0.9876 - loss: 0.3695 - val_accuracy: 1.0000 - val_loss: 0.3442
Epoch 60/100
40/40 - 0s - 6ms/step - accuracy: 0.9910 - loss: 0.3563 - val_accuracy: 1.0000 - val_loss: 0.3381
Epoch 61/100
40/40 - 0s - 6ms/step - accuracy: 0.9912 - loss: 0.3450 - val_accuracy: 1.0000 - val_loss: 0.3330
Epoch 62/100
40/40 - 0s - 6ms/step - accuracy: 0.9921 - loss: 0.3348 - val_accuracy: 1.0000 - val_loss: 0.3222
Epoch 63/100
40/40 - 0s - 6ms/step - accuracy: 0.9930 - loss: 0.3229 - val_accuracy: 1.0000 - val_loss: 0.3099
Epoch 64/100
40/40 - 0s - 6ms/step - accuracy: 0.9940 - loss: 0.3146 - val_accuracy: 1.0000 - val_loss: 0.3029
Epoch 65/100
40/40 - 0s - 6ms/step - accuracy: 0.9927 - loss: 0.3039 - val_accuracy: 1.0000 - val_loss: 0.2836
Epoch 66/100
40/40 - 0s - 6ms/step - accuracy: 0.9949 - loss: 0.2940 - val_accuracy: 1.0000 - val_loss: 0.2749
Epoch 67/100
40/40 - 0s - 6ms/step - accuracy: 0.9953 - loss: 0.2848 - val_accuracy: 1.0000 - val_loss: 0.2663
Epoch 68/100
40/40 - 0s - 6ms/step - accuracy: 0.9950 - loss: 0.2758 - val_accuracy: 1.0000 - val_loss: 0.2507
Epoch 69/100
40/40 - 0s - 6ms/step - accuracy: 0.9960 - loss: 0.2660 - val_accuracy: 1.0000 - val_loss: 0.2436
Epoch 70/100
40/40 - 0s - 6ms/step - accuracy: 0.9961 - loss: 0.2577 - val_accuracy: 1.0000 - val_loss: 0.2565
Epoch 71/100
40/40 - 0s - 6ms/step - accuracy: 0.9955 - loss: 0.2493 - val_accuracy: 1.0000 - val_loss: 0.2347
Epoch 72/100
40/40 - 0s - 6ms/step - accuracy: 0.9967 - loss: 0.2404 - val_accuracy: 1.0000 - val_loss: 0.2227
Epoch 73/100
40/40 - 0s - 6ms/step - accuracy: 0.9968 - loss: 0.2329 - val_accuracy: 1.0000 - val_loss: 0.2123
Epoch 74/100
40/40 - 0s - 6ms/step - accuracy: 0.9976 - loss: 0.2239 - val_accuracy: 1.0000 - val_loss: 0.2049
Epoch 75/100
40/40 - 0s - 6ms/step - accuracy: 0.9968 - loss: 0.2174 - val_accuracy: 1.0000 - val_loss: 0.1966
Epoch 76/100
40/40 - 0s - 6ms/step - accuracy: 0.9983 - loss: 0.2080 - val_accuracy: 1.0000 - val_loss: 0.1862
Epoch 77/100
40/40 - 0s - 6ms/step - accuracy: 0.9968 - loss: 0.2027 - val_accuracy: 1.0000 - val_loss: 0.1832
Epoch 78/100
40/40 - 0s - 6ms/step - accuracy: 0.9984 - loss: 0.1937 - val_accuracy: 1.0000 - val_loss: 0.1740
Epoch 79/100
40/40 - 0s - 6ms/step - accuracy: 0.9980 - loss: 0.1905 - val_accuracy: 1.0000 - val_loss: 0.1749
Epoch 80/100
40/40 - 0s - 6ms/step - accuracy: 0.9984 - loss: 0.1814 - val_accuracy: 1.0000 - val_loss: 0.1621
Epoch 81/100
40/40 - 0s - 6ms/step - accuracy: 0.9976 - loss: 0.1745 - val_accuracy: 1.0000 - val_loss: 0.1531
Epoch 82/100
40/40 - 0s - 6ms/step - accuracy: 0.9989 - loss: 0.1687 - val_accuracy: 1.0000 - val_loss: 0.1518
Epoch 83/100
40/40 - 0s - 6ms/step - accuracy: 0.9977 - loss: 0.1641 - val_accuracy: 1.0000 - val_loss: 0.1476
Epoch 84/100
40/40 - 0s - 6ms/step - accuracy: 0.9984 - loss: 0.1580 - val_accuracy: 1.0000 - val_loss: 0.1388
Epoch 85/100
40/40 - 0s - 6ms/step - accuracy: 0.9980 - loss: 0.1513 - val_accuracy: 1.0000 - val_loss: 0.1320
Epoch 86/100
40/40 - 0s - 6ms/step - accuracy: 0.9977 - loss: 0.1458 - val_accuracy: 1.0000 - val_loss: 0.1303
Epoch 87/100
40/40 - 0s - 6ms/step - accuracy: 0.9980 - loss: 0.1408 - val_accuracy: 1.0000 - val_loss: 0.1226
Epoch 88/100
40/40 - 0s - 6ms/step - accuracy: 0.9983 - loss: 0.1351 - val_accuracy: 1.0000 - val_loss: 0.1133
Epoch 89/100
40/40 - 0s - 6ms/step - accuracy: 0.9991 - loss: 0.1305 - val_accuracy: 1.0000 - val_loss: 0.1169
Epoch 90/100
40/40 - 0s - 6ms/step - accuracy: 0.9989 - loss: 0.1263 - val_accuracy: 1.0000 - val_loss: 0.1114
Epoch 91/100
40/40 - 0s - 6ms/step - accuracy: 0.9992 - loss: 0.1219 - val_accuracy: 1.0000 - val_loss: 0.1014
Epoch 92/100
40/40 - 0s - 6ms/step - accuracy: 0.9988 - loss: 0.1171 - val_accuracy: 1.0000 - val_loss: 0.0998
Epoch 93/100
40/40 - 0s - 6ms/step - accuracy: 0.9994 - loss: 0.1126 - val_accuracy: 1.0000 - val_loss: 0.0954
Epoch 94/100
40/40 - 0s - 6ms/step - accuracy: 0.9990 - loss: 0.1080 - val_accuracy: 1.0000 - val_loss: 0.0952
Epoch 95/100
40/40 - 0s - 6ms/step - accuracy: 0.9996 - loss: 0.1053 - val_accuracy: 1.0000 - val_loss: 0.0872
Epoch 96/100
40/40 - 0s - 6ms/step - accuracy: 0.9995 - loss: 0.1001 - val_accuracy: 1.0000 - val_loss: 0.0807
Epoch 97/100
40/40 - 0s - 6ms/step - accuracy: 0.9991 - loss: 0.0973 - val_accuracy: 1.0000 - val_loss: 0.0825
Epoch 98/100
40/40 - 0s - 6ms/step - accuracy: 0.9992 - loss: 0.0937 - val_accuracy: 1.0000 - val_loss: 0.0754
Epoch 99/100
40/40 - 0s - 6ms/step - accuracy: 0.9995 - loss: 0.0893 - val_accuracy: 1.0000 - val_loss: 0.0726
Epoch 100/100
40/40 - 0s - 6ms/step - accuracy: 0.9992 - loss: 0.0863 - val_accuracy: 1.0000 - val_loss: 0.0673
<keras.src.callbacks.history.History at 0x7f27a09a0440>
As we see, the network is essentially perfect now.