Keras and the Last Number Problem#
Let’s see if we can do better than our simple hidden layer NN with the last number problem.
import numpy as np
import keras
from keras.utils import np_utils
2025-03-08 16:42:59.409778: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-08 16:42:59.413059: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-08 16:42:59.421833: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1741452179.436133 4813 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1741452179.440318 4813 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-08 16:42:59.457226: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Cell In[1], line 3
1 import numpy as np
2 import keras
----> 3 from keras.utils import np_utils
ImportError: cannot import name 'np_utils' from 'keras.utils' (/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/keras/api/utils/__init__.py)
We’ll use the same data class
class ModelDataCategorical:
"""this is the model data for our "last number" training set. We
produce input of length N, consisting of numbers 0-9 and store
the result in a 10-element array as categorical data.
"""
def __init__(self, N=10):
self.N = N
# our model input data
self.x = np.random.randint(0, high=10, size=N)
self.x_scaled = self.x / 10 + 0.05
# our scaled model output data
self.y = np.array([self.x[-1]])
self.y_scaled = np.zeros(10) + 0.01
self.y_scaled[self.x[-1]] = 0.99
def interpret_result(self, out):
"""take the network output and return the number we predict"""
return np.argmax(out)
For Keras, we need to pack the scaled data (both input and output) into arrays. We’ll use
the Keras np_utils.to_categorical() to make the data categorical.
Let’s make both a training set and a test set
x_train = []
y_train = []
for _ in range(10000):
m = ModelDataCategorical()
x_train.append(m.x_scaled)
y_train.append(m.y)
x_train = np.asarray(x_train)
y_train = np_utils.to_categorical(y_train, 10)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[3], line 9
6 y_train.append(m.y)
8 x_train = np.asarray(x_train)
----> 9 y_train = np_utils.to_categorical(y_train, 10)
NameError: name 'np_utils' is not defined
x_test = []
y_test = []
for _ in range(1000):
m = ModelDataCategorical()
x_test.append(m.x_scaled)
y_test.append(m.y)
x_test = np.asarray(x_test)
y_test = np_utils.to_categorical(y_test, 10)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[4], line 9
6 y_test.append(m.y)
8 x_test = np.asarray(x_test)
----> 9 y_test = np_utils.to_categorical(y_test, 10)
NameError: name 'np_utils' is not defined
Check to make sure the data looks like we expect:
x_train[0]
array([0.85, 0.95, 0.05, 0.15, 0.85, 0.65, 0.75, 0.05, 0.35, 0.75])
y_train[0]
array([7])
Now let’s build our network. We’ll use just a single hidden layer, but instead of the sigmoid used before, we’ll use RELU and the softmax activations.
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from tensorflow.keras.optimizers import RMSprop
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 2
1 from keras.models import Sequential
----> 2 from keras.layers.core import Dense, Dropout, Activation
3 from tensorflow.keras.optimizers import RMSprop
ModuleNotFoundError: No module named 'keras.layers.core'
model = Sequential()
model.add(Dense(100, input_dim=10, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(10, activation="softmax"))
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[8], line 2
1 model = Sequential()
----> 2 model.add(Dense(100, input_dim=10, activation="relu"))
3 model.add(Dropout(0.1))
4 model.add(Dense(10, activation="softmax"))
NameError: name 'Dense' is not defined
rms = RMSprop()
model.compile(loss='categorical_crossentropy',
optimizer=rms, metrics=['accuracy'])
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[9], line 1
----> 1 rms = RMSprop()
2 model.compile(loss='categorical_crossentropy',
3 optimizer=rms, metrics=['accuracy'])
NameError: name 'RMSprop' is not defined
Now we can train and test each epoch to see how we do
epochs = 100
batch_size = 256
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
validation_data=(x_test, y_test), verbose=2)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[10], line 3
1 epochs = 100
2 batch_size = 256
----> 3 model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size,
4 validation_data=(x_test, y_test), verbose=2)
File /opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File /opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/keras/src/trainers/trainer.py:1049, in Trainer._assert_compile_called(self, method_name)
1047 else:
1048 msg += f"calling `{method_name}()`."
-> 1049 raise ValueError(msg)
ValueError: You must call `compile()` before using the model.
As we see, the network is essentially perfect now.