Our Galactic Dataset

Our Galactic Dataset#

Galaxies come in many different shapes. Large surveys can images 100,000s or millions of galaxies, and we want to be able to classify them into types. We can use a neural network for this.

We will use the Galaxy10 DECaLS dataset (Leung & Bovy 2024), which contains 17736 images of galaxies, each 256x256 pixels and 3 colors (g, r, and z bands), classified into 10 different categories. It also provides RA and dec, redshift, and pixel scale—we’ll ignore those.

Tip

We can create additional training data from this set via transformations: rotating it, flipping it, etc. That can help with the training. Since the orientation of the galaxies do not matter, this should not be a problem.

You can get the dataset from Zenodo: Galaxy10_DECals.h5

This is an HDF5 file—this is a self-describing binary format that is popular in scientific computing. We can read it in python using the h5py library.

Warning

This is a big dataset (~2.5 GB), so we need to be careful with memory.

Note

There is not a separate training and test set, so we will partition this dataset ourselves.

import numpy as np
import matplotlib.pyplot as plt
import h5py

If we look at the dataset, we can see the fields:

ds = h5py.File("Galaxy10_DECals.h5")
ds.keys()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[2], line 1
----> 1 ds = h5py.File("Galaxy10_DECals.h5")
      2 ds.keys()

File /opt/hostedtoolcache/Python/3.14.10/x64/lib/python3.14/site-packages/h5py/_hl/files.py:555, in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, track_times, **kwds)
    546     fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
    547                      locking, page_buf_size, min_meta_keep, min_raw_keep,
    548                      alignment_threshold=alignment_threshold,
    549                      alignment_interval=alignment_interval,
    550                      meta_block_size=meta_block_size,
    551                      **kwds)
    552     fcpl = make_fcpl(track_order=track_order, track_times=track_times,
    553                      fs_strategy=fs_strategy, fs_persist=fs_persist,
    554                      fs_threshold=fs_threshold, fs_page_size=fs_page_size)
--> 555     fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    557 if isinstance(libver, tuple):
    558     self._libver = libver

File /opt/hostedtoolcache/Python/3.14.10/x64/lib/python3.14/site-packages/h5py/_hl/files.py:232, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    230     if swmr:
    231         flags |= h5f.ACC_SWMR_READ
--> 232     fid = h5f.open(name, flags, fapl=fapl)
    233 elif mode == 'r+':
    234     fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()
---> 54 'Could not get source, probably due dynamically evaluated source code.'

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()
---> 55 'Could not get source, probably due dynamically evaluated source code.'

File h5py/h5f.pyx:106, in h5py.h5f.open()
--> 106 'Could not get source, probably due dynamically evaluated source code.'

FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = 'Galaxy10_DECals.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

let’s look at the answers

ans = np.array(ds["ans"])
ans.shape, ans.dtype
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 ans = np.array(ds["ans"])
      2 ans.shape, ans.dtype

NameError: name 'ds' is not defined

We see that there are 17736 records, stored as a NumPy array.

Important

The galaxies are stored in the dataset sorted by type. So all of the “disturbed galaxies” (type 0) come first, then the “merging galaxies”, …

Exploring the data#

Let’s define descriptive names for the galaxy types

galaxy_types = {0: "disturbed galaxies",
                1: "merging galaxies",
                2: "round smooth galaxies",
                3: "in-between round smooth galaxies",
                4: "cigar shaped smooth galaxies",
                5: "barred spiral galaxies",
                6: "unbarred tight spiral galaxies",
                7: "unbarred loose spiral galaxies",
                8: "edge-on galaxies without bulge",
                9: "edge-on galaxies with bulge"}

We’ll also create a simple class (Galaxy) to manage the data.

Here we pass in the image data for a single galaxy as a NumPy uint8 array—the same datatype used in the file. This class will convert it to a single precision floating-point array, scaled to fall between 0 and 1. We also convert the answer to a categorical datatype (and array of length 10).

class Galaxy:
    def __init__(self, data, answer, *, index=-1):
        self.data = np.array(data, dtype=np.float32) / 255.0 * 0.99 + 0.01
        self.answer = answer

        self.out = np.zeros(10, dtype=np.float32) + 0.01
        self.out[self.answer] = 0.99

        self.index = index

    def plot(self, ax=None):
        if ax is None:
            fig, ax = plt.subplots()
        ax.imshow(self.data, interpolation="nearest")
        ax.text(0.025, 0.95, f"answer: {self.answer}",
                color="white", transform=ax.transAxes)

    def validate(self, prediction):
        """check if a categorical prediction matches the answer"""
        return np.argmax(prediction) == self.answer

Now let’s find the first example of each type and plot it

images = ds["images"]
images.dtype
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 images = ds["images"]
      2 images.dtype

NameError: name 'ds' is not defined

Note

This doesn’t read in the data. It just provides a Dataset type that can be used to access the data, so we can get a single galaxy at a time.

from mpl_toolkits.axes_grid1 import ImageGrid
fig = plt.figure()
grid = ImageGrid(fig, 111,
                 nrows_ncols=(4, 3),
                 axes_pad=(0.25, 0.5))

for i in range(10):
    idx = np.where(ans == i)[0][0]
    g = Galaxy(images[idx, :, :, :], ans[idx])
    g.plot(ax=grid[i])
    grid[i].set_title(f"{idx}: {galaxy_types[i]}", fontsize="small")

grid[10].set_axis_off()
grid[11].set_axis_off()
fig.set_size_inches(10, 15)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 8
      4                  nrows_ncols=(4, 3),
      5                  axes_pad=(0.25, 0.5))
      6 
      7 for i in range(10):
----> 8     idx = np.where(ans == i)[0][0]
      9     g = Galaxy(images[idx, :, :, :], ans[idx])
     10     g.plot(ax=grid[i])
     11     grid[i].set_title(f"{idx}: {galaxy_types[i]}", fontsize="small")

NameError: name 'ans' is not defined
../_images/9c3ef144f9c4ee6f967ec03edf2ff7e89676d71b74b10ab690136b2e3ab0a89e.png

Important

Since each class of galaxy is stored together in the file, we will want to randomize the order before splitting into training and testing sets.

Convolution layers#

A convolution neural network takes a small group of pixels in the image (like \(3\times 3\)) and apply a kernel (or many kernels) to the data and connect it to the next layer. As a result, instead of every pixel on the input layer being connected to every neuron on the next layer, many fewer connections are created.

We won’t write this from scratch, but instead we will use Keras to implement it. Typically, a few convolution layers are stacked, with each using many kernels, and then a reduction or pooling is done, reducing the size of the data. Many such groups of convolution + pooling may be connected together before finally doing a dense connection to the output layer.