Keras and the Last Number Problem#
Let’s see if we can do better than our simple hidden layer NN with the last number problem.
import numpy as np
import keras
from keras.utils import to_categorical
2025-05-13 13:31:12.128631: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-13 13:31:12.131814: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-13 13:31:12.140304: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1747143072.154290 7387 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1747143072.158546 7387 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1747143072.169986 7387 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1747143072.169997 7387 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1747143072.169999 7387 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1747143072.170001 7387 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-05-13 13:31:12.174087: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
We’ll use the same data class
class ModelDataCategorical:
"""this is the model data for our "last number" training set. We
produce input of length N, consisting of numbers 0-9 and store
the result in a 10-element array as categorical data.
"""
def __init__(self, N=10):
self.N = N
# our model input data
self.x = np.random.randint(0, high=10, size=N)
self.x_scaled = self.x / 10 + 0.05
# our scaled model output data
self.y = np.array([self.x[-1]])
self.y_scaled = np.zeros(10) + 0.01
self.y_scaled[self.x[-1]] = 0.99
def interpret_result(self, out):
"""take the network output and return the number we predict"""
return np.argmax(out)
For Keras, we need to pack the scaled data (both input and output) into arrays. We’ll use
the Keras to_categorical()
to make the data categorical.
Let’s make both a training set and a test set
x_train = []
y_train = []
for _ in range(10000):
m = ModelDataCategorical()
x_train.append(m.x_scaled)
y_train.append(m.y)
x_train = np.asarray(x_train)
y_train = to_categorical(y_train, 10)
x_test = []
y_test = []
for _ in range(1000):
m = ModelDataCategorical()
x_test.append(m.x_scaled)
y_test.append(m.y)
x_test = np.asarray(x_test)
y_test = to_categorical(y_test, 10)
Check to make sure the data looks like we expect:
x_train[0]
array([0.95, 0.05, 0.35, 0.45, 0.35, 0.95, 0.65, 0.85, 0.35, 0.15])
y_train[0]
array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.])
Creating the network#
Now let’s build our network. We’ll use just a single hidden layer, but instead of the sigmoid used before, we’ll use RELU and the softmax activations.
from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation
from keras.optimizers import RMSprop
model = Sequential()
model.add(Input((10,)))
model.add(Dense(100, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(10, activation="softmax"))
2025-05-13 13:31:14.759455: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
rms = RMSprop()
model.compile(loss='categorical_crossentropy',
optimizer=rms, metrics=['accuracy'])
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 100) │ 1,100 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 100) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 10) │ 1,010 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,110 (8.24 KB)
Trainable params: 2,110 (8.24 KB)
Non-trainable params: 0 (0.00 B)
Now we have ~ 2k parameters to fit.
Training#
Now we can train and test each epoch to see how we do
epochs = 100
batch_size = 256
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
validation_data=(x_test, y_test), verbose=2)
Epoch 1/100
40/40 - 1s - 16ms/step - accuracy: 0.1452 - loss: 2.2612 - val_accuracy: 0.2200 - val_loss: 2.2045
Epoch 2/100
40/40 - 0s - 3ms/step - accuracy: 0.2134 - loss: 2.1592 - val_accuracy: 0.2500 - val_loss: 2.1104
Epoch 3/100
40/40 - 0s - 3ms/step - accuracy: 0.2621 - loss: 2.0611 - val_accuracy: 0.2760 - val_loss: 2.0036
Epoch 4/100
40/40 - 0s - 3ms/step - accuracy: 0.2881 - loss: 1.9643 - val_accuracy: 0.3200 - val_loss: 1.9099
Epoch 5/100
40/40 - 0s - 3ms/step - accuracy: 0.3182 - loss: 1.8732 - val_accuracy: 0.3250 - val_loss: 1.8200
Epoch 6/100
40/40 - 0s - 3ms/step - accuracy: 0.3448 - loss: 1.7877 - val_accuracy: 0.3330 - val_loss: 1.7377
Epoch 7/100
40/40 - 0s - 3ms/step - accuracy: 0.3738 - loss: 1.7112 - val_accuracy: 0.4100 - val_loss: 1.6659
Epoch 8/100
40/40 - 0s - 2ms/step - accuracy: 0.4096 - loss: 1.6372 - val_accuracy: 0.4300 - val_loss: 1.5951
Epoch 9/100
40/40 - 0s - 2ms/step - accuracy: 0.4215 - loss: 1.5761 - val_accuracy: 0.4790 - val_loss: 1.5325
Epoch 10/100
40/40 - 0s - 2ms/step - accuracy: 0.4547 - loss: 1.5137 - val_accuracy: 0.4500 - val_loss: 1.4779
Epoch 11/100
40/40 - 0s - 2ms/step - accuracy: 0.4714 - loss: 1.4626 - val_accuracy: 0.4720 - val_loss: 1.4305
Epoch 12/100
40/40 - 0s - 2ms/step - accuracy: 0.5063 - loss: 1.4073 - val_accuracy: 0.5490 - val_loss: 1.3743
Epoch 13/100
40/40 - 0s - 2ms/step - accuracy: 0.5292 - loss: 1.3641 - val_accuracy: 0.6310 - val_loss: 1.3264
Epoch 14/100
40/40 - 0s - 2ms/step - accuracy: 0.5520 - loss: 1.3172 - val_accuracy: 0.6530 - val_loss: 1.2794
Epoch 15/100
40/40 - 0s - 2ms/step - accuracy: 0.5774 - loss: 1.2747 - val_accuracy: 0.7000 - val_loss: 1.2467
Epoch 16/100
40/40 - 0s - 2ms/step - accuracy: 0.6002 - loss: 1.2380 - val_accuracy: 0.6200 - val_loss: 1.2057
Epoch 17/100
40/40 - 0s - 2ms/step - accuracy: 0.6111 - loss: 1.2004 - val_accuracy: 0.7140 - val_loss: 1.1654
Epoch 18/100
40/40 - 0s - 2ms/step - accuracy: 0.6391 - loss: 1.1667 - val_accuracy: 0.7340 - val_loss: 1.1350
Epoch 19/100
40/40 - 0s - 2ms/step - accuracy: 0.6576 - loss: 1.1326 - val_accuracy: 0.6840 - val_loss: 1.1117
Epoch 20/100
40/40 - 0s - 2ms/step - accuracy: 0.6653 - loss: 1.1052 - val_accuracy: 0.6250 - val_loss: 1.0750
Epoch 21/100
40/40 - 0s - 2ms/step - accuracy: 0.6772 - loss: 1.0771 - val_accuracy: 0.7740 - val_loss: 1.0406
Epoch 22/100
40/40 - 0s - 2ms/step - accuracy: 0.7009 - loss: 1.0457 - val_accuracy: 0.7760 - val_loss: 1.0145
Epoch 23/100
40/40 - 0s - 2ms/step - accuracy: 0.7148 - loss: 1.0186 - val_accuracy: 0.7180 - val_loss: 0.9933
Epoch 24/100
40/40 - 0s - 2ms/step - accuracy: 0.7188 - loss: 0.9930 - val_accuracy: 0.8550 - val_loss: 0.9640
Epoch 25/100
40/40 - 0s - 2ms/step - accuracy: 0.7397 - loss: 0.9695 - val_accuracy: 0.7810 - val_loss: 0.9491
Epoch 26/100
40/40 - 0s - 2ms/step - accuracy: 0.7536 - loss: 0.9444 - val_accuracy: 0.8700 - val_loss: 0.9138
Epoch 27/100
40/40 - 0s - 2ms/step - accuracy: 0.7682 - loss: 0.9223 - val_accuracy: 0.8170 - val_loss: 0.8914
Epoch 28/100
40/40 - 0s - 2ms/step - accuracy: 0.7752 - loss: 0.9008 - val_accuracy: 0.8690 - val_loss: 0.8679
Epoch 29/100
40/40 - 0s - 2ms/step - accuracy: 0.7924 - loss: 0.8785 - val_accuracy: 0.8820 - val_loss: 0.8460
Epoch 30/100
40/40 - 0s - 2ms/step - accuracy: 0.8079 - loss: 0.8538 - val_accuracy: 0.8460 - val_loss: 0.8230
Epoch 31/100
40/40 - 0s - 2ms/step - accuracy: 0.8088 - loss: 0.8344 - val_accuracy: 0.9020 - val_loss: 0.8008
Epoch 32/100
40/40 - 0s - 2ms/step - accuracy: 0.8224 - loss: 0.8121 - val_accuracy: 0.8640 - val_loss: 0.7856
Epoch 33/100
40/40 - 0s - 2ms/step - accuracy: 0.8376 - loss: 0.7907 - val_accuracy: 0.9010 - val_loss: 0.7588
Epoch 34/100
40/40 - 0s - 2ms/step - accuracy: 0.8442 - loss: 0.7732 - val_accuracy: 0.9170 - val_loss: 0.7438
Epoch 35/100
40/40 - 0s - 2ms/step - accuracy: 0.8526 - loss: 0.7537 - val_accuracy: 0.8740 - val_loss: 0.7342
Epoch 36/100
40/40 - 0s - 2ms/step - accuracy: 0.8558 - loss: 0.7345 - val_accuracy: 0.9040 - val_loss: 0.7063
Epoch 37/100
40/40 - 0s - 2ms/step - accuracy: 0.8658 - loss: 0.7179 - val_accuracy: 0.9310 - val_loss: 0.6880
Epoch 38/100
40/40 - 0s - 2ms/step - accuracy: 0.8681 - loss: 0.7021 - val_accuracy: 0.9170 - val_loss: 0.6764
Epoch 39/100
40/40 - 0s - 2ms/step - accuracy: 0.8829 - loss: 0.6825 - val_accuracy: 0.8950 - val_loss: 0.6573
Epoch 40/100
40/40 - 0s - 2ms/step - accuracy: 0.8847 - loss: 0.6632 - val_accuracy: 0.9370 - val_loss: 0.6351
Epoch 41/100
40/40 - 0s - 2ms/step - accuracy: 0.8927 - loss: 0.6471 - val_accuracy: 0.9670 - val_loss: 0.6127
Epoch 42/100
40/40 - 0s - 2ms/step - accuracy: 0.8998 - loss: 0.6304 - val_accuracy: 0.9590 - val_loss: 0.6024
Epoch 43/100
40/40 - 0s - 2ms/step - accuracy: 0.9051 - loss: 0.6152 - val_accuracy: 0.9580 - val_loss: 0.5850
Epoch 44/100
40/40 - 0s - 2ms/step - accuracy: 0.9169 - loss: 0.5980 - val_accuracy: 0.9470 - val_loss: 0.5724
Epoch 45/100
40/40 - 0s - 3ms/step - accuracy: 0.9175 - loss: 0.5813 - val_accuracy: 0.9490 - val_loss: 0.5584
Epoch 46/100
40/40 - 0s - 2ms/step - accuracy: 0.9213 - loss: 0.5677 - val_accuracy: 0.9800 - val_loss: 0.5342
Epoch 47/100
40/40 - 0s - 2ms/step - accuracy: 0.9275 - loss: 0.5520 - val_accuracy: 0.9650 - val_loss: 0.5253
Epoch 48/100
40/40 - 0s - 2ms/step - accuracy: 0.9303 - loss: 0.5385 - val_accuracy: 0.9730 - val_loss: 0.5141
Epoch 49/100
40/40 - 0s - 2ms/step - accuracy: 0.9403 - loss: 0.5240 - val_accuracy: 0.9740 - val_loss: 0.4963
Epoch 50/100
40/40 - 0s - 2ms/step - accuracy: 0.9457 - loss: 0.5098 - val_accuracy: 0.9900 - val_loss: 0.4862
Epoch 51/100
40/40 - 0s - 2ms/step - accuracy: 0.9474 - loss: 0.4950 - val_accuracy: 0.9790 - val_loss: 0.4706
Epoch 52/100
40/40 - 0s - 2ms/step - accuracy: 0.9536 - loss: 0.4824 - val_accuracy: 0.9590 - val_loss: 0.4611
Epoch 53/100
40/40 - 0s - 2ms/step - accuracy: 0.9570 - loss: 0.4699 - val_accuracy: 0.9950 - val_loss: 0.4468
Epoch 54/100
40/40 - 0s - 2ms/step - accuracy: 0.9610 - loss: 0.4566 - val_accuracy: 0.9840 - val_loss: 0.4335
Epoch 55/100
40/40 - 0s - 3ms/step - accuracy: 0.9651 - loss: 0.4459 - val_accuracy: 0.9990 - val_loss: 0.4169
Epoch 56/100
40/40 - 0s - 2ms/step - accuracy: 0.9662 - loss: 0.4324 - val_accuracy: 0.9810 - val_loss: 0.4204
Epoch 57/100
40/40 - 0s - 2ms/step - accuracy: 0.9702 - loss: 0.4217 - val_accuracy: 0.9820 - val_loss: 0.4072
Epoch 58/100
40/40 - 0s - 2ms/step - accuracy: 0.9728 - loss: 0.4089 - val_accuracy: 0.9990 - val_loss: 0.3815
Epoch 59/100
40/40 - 0s - 2ms/step - accuracy: 0.9744 - loss: 0.3981 - val_accuracy: 0.9990 - val_loss: 0.3701
Epoch 60/100
40/40 - 0s - 2ms/step - accuracy: 0.9762 - loss: 0.3883 - val_accuracy: 0.9990 - val_loss: 0.3638
Epoch 61/100
40/40 - 0s - 2ms/step - accuracy: 0.9787 - loss: 0.3772 - val_accuracy: 1.0000 - val_loss: 0.3487
Epoch 62/100
40/40 - 0s - 2ms/step - accuracy: 0.9809 - loss: 0.3658 - val_accuracy: 1.0000 - val_loss: 0.3356
Epoch 63/100
40/40 - 0s - 2ms/step - accuracy: 0.9816 - loss: 0.3540 - val_accuracy: 1.0000 - val_loss: 0.3298
Epoch 64/100
40/40 - 0s - 2ms/step - accuracy: 0.9831 - loss: 0.3444 - val_accuracy: 1.0000 - val_loss: 0.3216
Epoch 65/100
40/40 - 0s - 3ms/step - accuracy: 0.9836 - loss: 0.3379 - val_accuracy: 0.9990 - val_loss: 0.3165
Epoch 66/100
40/40 - 0s - 2ms/step - accuracy: 0.9855 - loss: 0.3273 - val_accuracy: 1.0000 - val_loss: 0.2992
Epoch 67/100
40/40 - 0s - 2ms/step - accuracy: 0.9893 - loss: 0.3171 - val_accuracy: 1.0000 - val_loss: 0.2898
Epoch 68/100
40/40 - 0s - 2ms/step - accuracy: 0.9902 - loss: 0.3081 - val_accuracy: 1.0000 - val_loss: 0.2809
Epoch 69/100
40/40 - 0s - 2ms/step - accuracy: 0.9901 - loss: 0.2984 - val_accuracy: 1.0000 - val_loss: 0.2729
Epoch 70/100
40/40 - 0s - 2ms/step - accuracy: 0.9898 - loss: 0.2904 - val_accuracy: 1.0000 - val_loss: 0.2700
Epoch 71/100
40/40 - 0s - 2ms/step - accuracy: 0.9911 - loss: 0.2816 - val_accuracy: 1.0000 - val_loss: 0.2642
Epoch 72/100
40/40 - 0s - 2ms/step - accuracy: 0.9924 - loss: 0.2746 - val_accuracy: 1.0000 - val_loss: 0.2481
Epoch 73/100
40/40 - 0s - 2ms/step - accuracy: 0.9934 - loss: 0.2659 - val_accuracy: 1.0000 - val_loss: 0.2422
Epoch 74/100
40/40 - 0s - 2ms/step - accuracy: 0.9921 - loss: 0.2571 - val_accuracy: 1.0000 - val_loss: 0.2314
Epoch 75/100
40/40 - 0s - 3ms/step - accuracy: 0.9920 - loss: 0.2499 - val_accuracy: 1.0000 - val_loss: 0.2265
Epoch 76/100
40/40 - 0s - 2ms/step - accuracy: 0.9956 - loss: 0.2406 - val_accuracy: 1.0000 - val_loss: 0.2173
Epoch 77/100
40/40 - 0s - 2ms/step - accuracy: 0.9951 - loss: 0.2333 - val_accuracy: 1.0000 - val_loss: 0.2149
Epoch 78/100
40/40 - 0s - 2ms/step - accuracy: 0.9954 - loss: 0.2266 - val_accuracy: 1.0000 - val_loss: 0.2065
Epoch 79/100
40/40 - 0s - 2ms/step - accuracy: 0.9958 - loss: 0.2189 - val_accuracy: 1.0000 - val_loss: 0.1948
Epoch 80/100
40/40 - 0s - 2ms/step - accuracy: 0.9954 - loss: 0.2121 - val_accuracy: 1.0000 - val_loss: 0.1899
Epoch 81/100
40/40 - 0s - 2ms/step - accuracy: 0.9956 - loss: 0.2051 - val_accuracy: 1.0000 - val_loss: 0.1885
Epoch 82/100
40/40 - 0s - 2ms/step - accuracy: 0.9963 - loss: 0.1996 - val_accuracy: 1.0000 - val_loss: 0.1725
Epoch 83/100
40/40 - 0s - 2ms/step - accuracy: 0.9966 - loss: 0.1910 - val_accuracy: 0.9990 - val_loss: 0.1876
Epoch 84/100
40/40 - 0s - 2ms/step - accuracy: 0.9968 - loss: 0.1860 - val_accuracy: 1.0000 - val_loss: 0.1642
Epoch 85/100
40/40 - 0s - 3ms/step - accuracy: 0.9972 - loss: 0.1811 - val_accuracy: 1.0000 - val_loss: 0.1637
Epoch 86/100
40/40 - 0s - 2ms/step - accuracy: 0.9965 - loss: 0.1730 - val_accuracy: 1.0000 - val_loss: 0.1520
Epoch 87/100
40/40 - 0s - 2ms/step - accuracy: 0.9974 - loss: 0.1688 - val_accuracy: 1.0000 - val_loss: 0.1492
Epoch 88/100
40/40 - 0s - 2ms/step - accuracy: 0.9972 - loss: 0.1642 - val_accuracy: 1.0000 - val_loss: 0.1437
Epoch 89/100
40/40 - 0s - 2ms/step - accuracy: 0.9985 - loss: 0.1568 - val_accuracy: 1.0000 - val_loss: 0.1380
Epoch 90/100
40/40 - 0s - 2ms/step - accuracy: 0.9981 - loss: 0.1520 - val_accuracy: 1.0000 - val_loss: 0.1278
Epoch 91/100
40/40 - 0s - 2ms/step - accuracy: 0.9980 - loss: 0.1463 - val_accuracy: 1.0000 - val_loss: 0.1240
Epoch 92/100
40/40 - 0s - 2ms/step - accuracy: 0.9978 - loss: 0.1414 - val_accuracy: 1.0000 - val_loss: 0.1173
Epoch 93/100
40/40 - 0s - 2ms/step - accuracy: 0.9989 - loss: 0.1368 - val_accuracy: 1.0000 - val_loss: 0.1138
Epoch 94/100
40/40 - 0s - 2ms/step - accuracy: 0.9987 - loss: 0.1308 - val_accuracy: 1.0000 - val_loss: 0.1108
Epoch 95/100
40/40 - 0s - 3ms/step - accuracy: 0.9985 - loss: 0.1271 - val_accuracy: 1.0000 - val_loss: 0.1067
Epoch 96/100
40/40 - 0s - 2ms/step - accuracy: 0.9987 - loss: 0.1232 - val_accuracy: 1.0000 - val_loss: 0.1056
Epoch 97/100
40/40 - 0s - 2ms/step - accuracy: 0.9988 - loss: 0.1184 - val_accuracy: 1.0000 - val_loss: 0.1028
Epoch 98/100
40/40 - 0s - 2ms/step - accuracy: 0.9991 - loss: 0.1134 - val_accuracy: 1.0000 - val_loss: 0.0926
Epoch 99/100
40/40 - 0s - 2ms/step - accuracy: 0.9987 - loss: 0.1102 - val_accuracy: 1.0000 - val_loss: 0.0895
Epoch 100/100
40/40 - 0s - 3ms/step - accuracy: 0.9990 - loss: 0.1055 - val_accuracy: 1.0000 - val_loss: 0.0847
<keras.src.callbacks.history.History at 0x7f12b3ab2690>
As we see, the network is essentially perfect now.