Keras and the Last Number Problem

Let’s see if we can do better than our simple hidden layer NN with the last number problem.

import numpy as np
import keras
from keras.utils import to_categorical

2025-04-18 16:21:17.351727: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-04-18 16:21:17.355099: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-04-18 16:21:17.363728: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1744993277.377662    6132 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1744993277.381928    6132 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1744993277.393950    6132 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1744993277.393965    6132 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1744993277.393967    6132 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1744993277.393969    6132 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-04-18 16:21:17.397991: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

We’ll use the same data class

class ModelDataCategorical:
    """this is the model data for our "last number" training set.  We
    produce input of length N, consisting of numbers 0-9 and store
    the result in a 10-element array as categorical data.

    """
    def __init__(self, N=10):
        self.N = N
        
        # our model input data
        self.x = np.random.randint(0, high=10, size=N)
        self.x_scaled = self.x / 10 + 0.05
        
        # our scaled model output data
        self.y = np.array([self.x[-1]])
        self.y_scaled = np.zeros(10) + 0.01
        self.y_scaled[self.x[-1]] = 0.99
        
    def interpret_result(self, out):
        """take the network output and return the number we predict"""
        return np.argmax(out)

For Keras, we need to pack the scaled data (both input and output) into arrays. We’ll use the Keras to_categorical() to make the data categorical.

Let’s make both a training set and a test set

x_train = []
y_train = []
for _ in range(10000):
    m = ModelDataCategorical()
    x_train.append(m.x_scaled)
    y_train.append(m.y)

x_train = np.asarray(x_train)
y_train = to_categorical(y_train, 10)

x_test = []
y_test = []
for _ in range(1000):
    m = ModelDataCategorical()
    x_test.append(m.x_scaled)
    y_test.append(m.y)

x_test = np.asarray(x_test)
y_test = to_categorical(y_test, 10)

Check to make sure the data looks like we expect:

x_train[0]

array([0.55, 0.55, 0.35, 0.15, 0.05, 0.65, 0.75, 0.25, 0.05, 0.95])

y_train[0]

array([0., 0., 0., 0., 0., 0., 0., 0., 0., 1.])

Now let’s build our network. We’ll use just a single hidden layer, but instead of the sigmoid used before, we’ll use RELU and the softmax activations.

from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation
from keras.optimizers import RMSprop

model = Sequential()
model.add(Input((10,)))
model.add(Dense(100, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(10, activation="softmax"))

2025-04-18 16:21:20.103283: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)

rms = RMSprop()
model.compile(loss='categorical_crossentropy',
              optimizer=rms, metrics=['accuracy'])

Now we can train and test each epoch to see how we do

epochs = 100
batch_size = 256
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
          validation_data=(x_test, y_test), verbose=2)

Epoch 1/100

40/40 - 1s - 17ms/step - accuracy: 0.1681 - loss: 2.2577 - val_accuracy: 0.2520 - val_loss: 2.1952

Epoch 2/100

40/40 - 0s - 3ms/step - accuracy: 0.2609 - loss: 2.1512 - val_accuracy: 0.2890 - val_loss: 2.0923

Epoch 3/100

40/40 - 0s - 3ms/step - accuracy: 0.2820 - loss: 2.0466 - val_accuracy: 0.3000 - val_loss: 1.9778

Epoch 4/100

40/40 - 0s - 3ms/step - accuracy: 0.3028 - loss: 1.9431 - val_accuracy: 0.3330 - val_loss: 1.8809

Epoch 5/100

40/40 - 0s - 3ms/step - accuracy: 0.3287 - loss: 1.8477 - val_accuracy: 0.3590 - val_loss: 1.7859

Epoch 6/100

40/40 - 0s - 3ms/step - accuracy: 0.3593 - loss: 1.7599 - val_accuracy: 0.3650 - val_loss: 1.7045

Epoch 7/100

40/40 - 0s - 3ms/step - accuracy: 0.3817 - loss: 1.6795 - val_accuracy: 0.4050 - val_loss: 1.6225

Epoch 8/100

40/40 - 0s - 3ms/step - accuracy: 0.4153 - loss: 1.6069 - val_accuracy: 0.4320 - val_loss: 1.5561

Epoch 9/100

40/40 - 0s - 3ms/step - accuracy: 0.4341 - loss: 1.5449 - val_accuracy: 0.5060 - val_loss: 1.4908

Epoch 10/100

40/40 - 0s - 2ms/step - accuracy: 0.4616 - loss: 1.4859 - val_accuracy: 0.4490 - val_loss: 1.4394

Epoch 11/100

40/40 - 0s - 2ms/step - accuracy: 0.4846 - loss: 1.4329 - val_accuracy: 0.5960 - val_loss: 1.3857

Epoch 12/100

40/40 - 0s - 2ms/step - accuracy: 0.5134 - loss: 1.3807 - val_accuracy: 0.5330 - val_loss: 1.3386

Epoch 13/100

40/40 - 0s - 2ms/step - accuracy: 0.5268 - loss: 1.3364 - val_accuracy: 0.5540 - val_loss: 1.2910

Epoch 14/100

40/40 - 0s - 3ms/step - accuracy: 0.5407 - loss: 1.2931 - val_accuracy: 0.6130 - val_loss: 1.2546

Epoch 15/100

40/40 - 0s - 2ms/step - accuracy: 0.5756 - loss: 1.2522 - val_accuracy: 0.5640 - val_loss: 1.2183

Epoch 16/100

40/40 - 0s - 2ms/step - accuracy: 0.5789 - loss: 1.2156 - val_accuracy: 0.5800 - val_loss: 1.1776

Epoch 17/100

40/40 - 0s - 2ms/step - accuracy: 0.5960 - loss: 1.1811 - val_accuracy: 0.6980 - val_loss: 1.1395

Epoch 18/100

40/40 - 0s - 3ms/step - accuracy: 0.6190 - loss: 1.1477 - val_accuracy: 0.6990 - val_loss: 1.1082

Epoch 19/100

40/40 - 0s - 3ms/step - accuracy: 0.6344 - loss: 1.1135 - val_accuracy: 0.6600 - val_loss: 1.0783

Epoch 20/100

40/40 - 0s - 3ms/step - accuracy: 0.6525 - loss: 1.0820 - val_accuracy: 0.6580 - val_loss: 1.0584

Epoch 21/100

40/40 - 0s - 3ms/step - accuracy: 0.6700 - loss: 1.0538 - val_accuracy: 0.7580 - val_loss: 1.0157

Epoch 22/100

40/40 - 0s - 2ms/step - accuracy: 0.6910 - loss: 1.0221 - val_accuracy: 0.6800 - val_loss: 0.9911

Epoch 23/100

40/40 - 0s - 2ms/step - accuracy: 0.6999 - loss: 1.0006 - val_accuracy: 0.7900 - val_loss: 0.9588

Epoch 24/100

40/40 - 0s - 3ms/step - accuracy: 0.7231 - loss: 0.9711 - val_accuracy: 0.7430 - val_loss: 0.9419

Epoch 25/100

40/40 - 0s - 2ms/step - accuracy: 0.7313 - loss: 0.9471 - val_accuracy: 0.7820 - val_loss: 0.9173

Epoch 26/100

40/40 - 0s - 2ms/step - accuracy: 0.7481 - loss: 0.9220 - val_accuracy: 0.7250 - val_loss: 0.8954

Epoch 27/100

40/40 - 0s - 3ms/step - accuracy: 0.7630 - loss: 0.8985 - val_accuracy: 0.7790 - val_loss: 0.8698

Epoch 28/100

40/40 - 0s - 3ms/step - accuracy: 0.7783 - loss: 0.8763 - val_accuracy: 0.8620 - val_loss: 0.8449

Epoch 29/100

40/40 - 0s - 2ms/step - accuracy: 0.7975 - loss: 0.8515 - val_accuracy: 0.7700 - val_loss: 0.8373

Epoch 30/100

40/40 - 0s - 2ms/step - accuracy: 0.8012 - loss: 0.8308 - val_accuracy: 0.8800 - val_loss: 0.7905

Epoch 31/100

40/40 - 0s - 3ms/step - accuracy: 0.8167 - loss: 0.8101 - val_accuracy: 0.8670 - val_loss: 0.7781

Epoch 32/100

40/40 - 0s - 2ms/step - accuracy: 0.8335 - loss: 0.7861 - val_accuracy: 0.8860 - val_loss: 0.7557

Epoch 33/100

40/40 - 0s - 2ms/step - accuracy: 0.8480 - loss: 0.7677 - val_accuracy: 0.9190 - val_loss: 0.7320

Epoch 34/100

40/40 - 0s - 3ms/step - accuracy: 0.8657 - loss: 0.7461 - val_accuracy: 0.9050 - val_loss: 0.7131

Epoch 35/100

40/40 - 0s - 2ms/step - accuracy: 0.8655 - loss: 0.7254 - val_accuracy: 0.9470 - val_loss: 0.6928

Epoch 36/100

40/40 - 0s - 3ms/step - accuracy: 0.8791 - loss: 0.7043 - val_accuracy: 0.9100 - val_loss: 0.6710

Epoch 37/100

40/40 - 0s - 3ms/step - accuracy: 0.8903 - loss: 0.6853 - val_accuracy: 0.9330 - val_loss: 0.6539

Epoch 38/100

40/40 - 0s - 2ms/step - accuracy: 0.9012 - loss: 0.6683 - val_accuracy: 0.9310 - val_loss: 0.6392

Epoch 39/100

40/40 - 0s - 2ms/step - accuracy: 0.9081 - loss: 0.6474 - val_accuracy: 0.9500 - val_loss: 0.6150

Epoch 40/100

40/40 - 0s - 3ms/step - accuracy: 0.9219 - loss: 0.6290 - val_accuracy: 0.9620 - val_loss: 0.6049

Epoch 41/100

40/40 - 0s - 2ms/step - accuracy: 0.9276 - loss: 0.6106 - val_accuracy: 0.9790 - val_loss: 0.5847

Epoch 42/100

40/40 - 0s - 2ms/step - accuracy: 0.9308 - loss: 0.5962 - val_accuracy: 0.9420 - val_loss: 0.5764

Epoch 43/100

40/40 - 0s - 2ms/step - accuracy: 0.9407 - loss: 0.5773 - val_accuracy: 0.9870 - val_loss: 0.5460

Epoch 44/100

40/40 - 0s - 2ms/step - accuracy: 0.9440 - loss: 0.5623 - val_accuracy: 0.9580 - val_loss: 0.5348

Epoch 45/100

40/40 - 0s - 3ms/step - accuracy: 0.9502 - loss: 0.5449 - val_accuracy: 0.9240 - val_loss: 0.5402

Epoch 46/100

40/40 - 0s - 2ms/step - accuracy: 0.9543 - loss: 0.5314 - val_accuracy: 0.9910 - val_loss: 0.4966

Epoch 47/100

40/40 - 0s - 3ms/step - accuracy: 0.9605 - loss: 0.5157 - val_accuracy: 0.9980 - val_loss: 0.4884

Epoch 48/100

40/40 - 0s - 3ms/step - accuracy: 0.9633 - loss: 0.5001 - val_accuracy: 0.9870 - val_loss: 0.4778

Epoch 49/100

40/40 - 0s - 3ms/step - accuracy: 0.9698 - loss: 0.4842 - val_accuracy: 0.9850 - val_loss: 0.4718

Epoch 50/100

40/40 - 0s - 3ms/step - accuracy: 0.9705 - loss: 0.4728 - val_accuracy: 1.0000 - val_loss: 0.4398

Epoch 51/100

40/40 - 0s - 2ms/step - accuracy: 0.9780 - loss: 0.4575 - val_accuracy: 0.9900 - val_loss: 0.4264

Epoch 52/100

40/40 - 0s - 3ms/step - accuracy: 0.9761 - loss: 0.4445 - val_accuracy: 0.9990 - val_loss: 0.4220

Epoch 53/100

40/40 - 0s - 3ms/step - accuracy: 0.9820 - loss: 0.4303 - val_accuracy: 1.0000 - val_loss: 0.4002

Epoch 54/100

40/40 - 0s - 3ms/step - accuracy: 0.9834 - loss: 0.4196 - val_accuracy: 1.0000 - val_loss: 0.3887

Epoch 55/100

40/40 - 0s - 3ms/step - accuracy: 0.9838 - loss: 0.4058 - val_accuracy: 0.9990 - val_loss: 0.3765

Epoch 56/100

40/40 - 0s - 3ms/step - accuracy: 0.9869 - loss: 0.3925 - val_accuracy: 1.0000 - val_loss: 0.3711

Epoch 57/100

40/40 - 0s - 3ms/step - accuracy: 0.9864 - loss: 0.3815 - val_accuracy: 1.0000 - val_loss: 0.3561

Epoch 58/100

40/40 - 0s - 2ms/step - accuracy: 0.9903 - loss: 0.3694 - val_accuracy: 1.0000 - val_loss: 0.3443

Epoch 59/100

40/40 - 0s - 3ms/step - accuracy: 0.9896 - loss: 0.3564 - val_accuracy: 1.0000 - val_loss: 0.3383

Epoch 60/100

40/40 - 0s - 3ms/step - accuracy: 0.9915 - loss: 0.3470 - val_accuracy: 1.0000 - val_loss: 0.3184

Epoch 61/100

40/40 - 0s - 3ms/step - accuracy: 0.9920 - loss: 0.3348 - val_accuracy: 1.0000 - val_loss: 0.3066

Epoch 62/100

40/40 - 0s - 3ms/step - accuracy: 0.9916 - loss: 0.3237 - val_accuracy: 1.0000 - val_loss: 0.3048

Epoch 63/100

40/40 - 0s - 3ms/step - accuracy: 0.9933 - loss: 0.3147 - val_accuracy: 1.0000 - val_loss: 0.2911

Epoch 64/100

40/40 - 0s - 3ms/step - accuracy: 0.9938 - loss: 0.3040 - val_accuracy: 1.0000 - val_loss: 0.2794

Epoch 65/100

40/40 - 0s - 3ms/step - accuracy: 0.9936 - loss: 0.2939 - val_accuracy: 1.0000 - val_loss: 0.2721

Epoch 66/100

40/40 - 0s - 3ms/step - accuracy: 0.9941 - loss: 0.2829 - val_accuracy: 1.0000 - val_loss: 0.2634

Epoch 67/100

40/40 - 0s - 3ms/step - accuracy: 0.9963 - loss: 0.2733 - val_accuracy: 1.0000 - val_loss: 0.2475

Epoch 68/100

40/40 - 0s - 3ms/step - accuracy: 0.9943 - loss: 0.2646 - val_accuracy: 1.0000 - val_loss: 0.2414

Epoch 69/100

40/40 - 0s - 3ms/step - accuracy: 0.9953 - loss: 0.2565 - val_accuracy: 1.0000 - val_loss: 0.2299

Epoch 70/100

40/40 - 0s - 3ms/step - accuracy: 0.9963 - loss: 0.2473 - val_accuracy: 1.0000 - val_loss: 0.2326

Epoch 71/100

40/40 - 0s - 3ms/step - accuracy: 0.9957 - loss: 0.2398 - val_accuracy: 1.0000 - val_loss: 0.2113

Epoch 72/100

40/40 - 0s - 3ms/step - accuracy: 0.9973 - loss: 0.2309 - val_accuracy: 1.0000 - val_loss: 0.2129

Epoch 73/100

40/40 - 0s - 2ms/step - accuracy: 0.9966 - loss: 0.2215 - val_accuracy: 1.0000 - val_loss: 0.2048

Epoch 74/100

40/40 - 0s - 2ms/step - accuracy: 0.9979 - loss: 0.2141 - val_accuracy: 1.0000 - val_loss: 0.1885

Epoch 75/100

40/40 - 0s - 2ms/step - accuracy: 0.9971 - loss: 0.2069 - val_accuracy: 1.0000 - val_loss: 0.1837

Epoch 76/100

40/40 - 0s - 3ms/step - accuracy: 0.9974 - loss: 0.1984 - val_accuracy: 1.0000 - val_loss: 0.1754

Epoch 77/100

40/40 - 0s - 2ms/step - accuracy: 0.9989 - loss: 0.1912 - val_accuracy: 1.0000 - val_loss: 0.1758

Epoch 78/100

40/40 - 0s - 2ms/step - accuracy: 0.9978 - loss: 0.1847 - val_accuracy: 1.0000 - val_loss: 0.1669

Epoch 79/100

40/40 - 0s - 3ms/step - accuracy: 0.9980 - loss: 0.1791 - val_accuracy: 1.0000 - val_loss: 0.1621

Epoch 80/100

40/40 - 0s - 3ms/step - accuracy: 0.9984 - loss: 0.1725 - val_accuracy: 1.0000 - val_loss: 0.1571

Epoch 81/100

40/40 - 0s - 2ms/step - accuracy: 0.9982 - loss: 0.1671 - val_accuracy: 1.0000 - val_loss: 0.1450

Epoch 82/100

40/40 - 0s - 3ms/step - accuracy: 0.9978 - loss: 0.1610 - val_accuracy: 1.0000 - val_loss: 0.1360

Epoch 83/100

40/40 - 0s - 2ms/step - accuracy: 0.9983 - loss: 0.1551 - val_accuracy: 1.0000 - val_loss: 0.1349

Epoch 84/100

40/40 - 0s - 2ms/step - accuracy: 0.9985 - loss: 0.1494 - val_accuracy: 1.0000 - val_loss: 0.1336

Epoch 85/100

40/40 - 0s - 2ms/step - accuracy: 0.9990 - loss: 0.1434 - val_accuracy: 1.0000 - val_loss: 0.1205

Epoch 86/100

40/40 - 0s - 3ms/step - accuracy: 0.9986 - loss: 0.1381 - val_accuracy: 1.0000 - val_loss: 0.1139

Epoch 87/100

40/40 - 0s - 2ms/step - accuracy: 0.9990 - loss: 0.1322 - val_accuracy: 1.0000 - val_loss: 0.1145

Epoch 88/100

40/40 - 0s - 2ms/step - accuracy: 0.9991 - loss: 0.1283 - val_accuracy: 1.0000 - val_loss: 0.1046

Epoch 89/100

40/40 - 0s - 3ms/step - accuracy: 0.9993 - loss: 0.1228 - val_accuracy: 1.0000 - val_loss: 0.1065

Epoch 90/100

40/40 - 0s - 3ms/step - accuracy: 0.9988 - loss: 0.1192 - val_accuracy: 1.0000 - val_loss: 0.1035

Epoch 91/100

40/40 - 0s - 3ms/step - accuracy: 0.9990 - loss: 0.1141 - val_accuracy: 1.0000 - val_loss: 0.0974

Epoch 92/100

40/40 - 0s - 3ms/step - accuracy: 0.9992 - loss: 0.1108 - val_accuracy: 1.0000 - val_loss: 0.0895

Epoch 93/100

40/40 - 0s - 2ms/step - accuracy: 0.9995 - loss: 0.1055 - val_accuracy: 1.0000 - val_loss: 0.0867

Epoch 94/100

40/40 - 0s - 3ms/step - accuracy: 0.9992 - loss: 0.1018 - val_accuracy: 1.0000 - val_loss: 0.0798

Epoch 95/100

40/40 - 0s - 3ms/step - accuracy: 0.9993 - loss: 0.0969 - val_accuracy: 1.0000 - val_loss: 0.0843

Epoch 96/100

40/40 - 0s - 3ms/step - accuracy: 0.9990 - loss: 0.0949 - val_accuracy: 1.0000 - val_loss: 0.0821

Epoch 97/100

40/40 - 0s - 3ms/step - accuracy: 0.9994 - loss: 0.0910 - val_accuracy: 1.0000 - val_loss: 0.0759

Epoch 98/100

40/40 - 0s - 3ms/step - accuracy: 0.9994 - loss: 0.0867 - val_accuracy: 1.0000 - val_loss: 0.0718

Epoch 99/100

40/40 - 0s - 3ms/step - accuracy: 0.9989 - loss: 0.0846 - val_accuracy: 1.0000 - val_loss: 0.0637

Epoch 100/100

40/40 - 0s - 3ms/step - accuracy: 0.9993 - loss: 0.0813 - val_accuracy: 1.0000 - val_loss: 0.0624

<keras.src.callbacks.history.History at 0x7f8194b53ed0>

As we see, the network is essentially perfect now.