Auto-differentiation#

Quite often we need the derivative of a function we are working with. If our function is very complex, it may not be possible to differentiate it analytically. This is where automatic differentiation or “autodiff” can help.

Consider the following function, which is a (crude) approximation to the equation of state for a degenerate gas + ideal gas nuclei (we used this in Application: One-zone X-ray burst):

import numpy as np

First we’ll define the necessary fundamental constants:

k_B = 1.38e-16  # erg / K
h = 6.67e-27  # erg s
m_e = 9.11e-28  # g
m_u = 1.66e-24  # g

# electron degeneracy
K = h**2 /(20 * m_e) * (3.0 / np.pi)**(2./3.) * m_u**(-5./3.)
    
# we'll use pure He for the compoosition
mu_I = 4.0
mu_e = 2.0

Now our function:

def pressure(rho, T):

    # ions are ideal gas
    p_ion = rho * k_B * T / (mu_I * m_u)

    # electrons can be ideal gas or degenerate -- we use an approximation to blend
    # the two regimes
    p_e_ideal = rho * k_B * T / (mu_e * m_u)
    p_e_deg = K * (rho / mu_e)**(5./3.)
    p_e = np.sqrt(p_e_ideal**2 + p_e_deg**2)

    return p_ion + p_e

Given a \(\rho, T\), we can get the total pressure. But what if we wanted \(\partial p/\partial \rho |_T\) or \(\partial p/\partial T|_\rho\)?

Finite-difference method#

We could use a finite-difference, like we did in Differentiation.

First, let’s define the thermodynamic state:

rho = 1.e5
T = 1.e7

Now we can get the pressure

p = pressure(rho, T)

We can use a centered-difference to get the derivative:

\[\left . \frac{dp}{d\rho} \right |_T \approx \frac{p(\rho + \Delta \rho, T) - p(\rho - \Delta\rho, T)}{2 \Delta\rho}\]
eps = 1.e-8
drho = rho * eps

dpdrho = (pressure(rho+drho, T) - pressure(rho-drho, T)) / (2.0 * drho)
print(f"p = {p:10.5g}; ∂p/∂ρ |_T = {dpdrho:10.5g}")
p = 7.1246e+20; ∂p/∂ρ |_T = 1.1719e+16

Note

For complex functions, a finite-difference can be slow. It can also suffer from accumulation of round-off error and result in an inaccurate derivative.

Introduction to automatic differentiation#

The basic idea of autodiff is to define a new datatype that carries both a value and a derivative and understands how mathematical operations propagate the derivative. We call this new type dual numbers or a “dual”.

When we work with a dual, it essentially just uses the chain rule automatically for us.

Note

We will implement “forward mode” autodiff, which works well when we are only differentiating with respect to a few variables.

Basic arithmetic operators#

For a quantity \(x\), we replace it with \(x + x'\epsilon\), where \(\epsilon\) is a special mathematical quantity (a formal infinitesimal) with the property \(\epsilon^2 = 0\). The coefficient \(x'\) carries the derivative with respect to the independent variable.

With this definition, we have:

  • addition: \((x + x'\epsilon) + (y + y'\epsilon) = x + y + (x' + y')\epsilon\)

  • subtraction: \((x + x'\epsilon) - (y + y'\epsilon) = x - y + (x' - y')\epsilon\)

  • multiplication: \((x + x'\epsilon) \cdot (y + y'\epsilon) = xy + (xy' + yx')\epsilon\)

  • division:

    \[\frac{x + x'\epsilon}{y + y'\epsilon} = \frac{\frac{x}{y} + \frac{x'\epsilon}{y}}{1 + \frac{y'\epsilon}{y}} = \left (\frac{x}{y} + \frac{x'\epsilon}{y} \right ) \left (1 - \frac{y'\epsilon}{y} \right ) = \frac{x}{y} + \left (\frac{x'y - xy'}{y^2} \right )\epsilon\]

We see that these just give the expression from calculus that we are familiar with.

Complex mathematical operations#

What about complex functions, like \(x^y\)? We can use the same ideas.

Start with:

\[q = (x + x'\epsilon)^{(y + y'\epsilon)}\]

It’s easiest to take the log first

Important

This assumes that \(x > 0\)

\[\begin{split} \log q &= (y + y'\epsilon) \log(x + x'\epsilon) \\ &= (y + y'\epsilon) \log \left [ x \left (1 + \frac{x'\epsilon}{x} \right ) \right ] \\ &= (y + y'\epsilon) \left [\log x + \underbrace{\log \left ( 1 + \frac{x'\epsilon}{x} \right )}_{= x'\epsilon/x} \right ] \\ &= y\log x + y'\epsilon \log x + \frac{yx'}{x} \epsilon \\ &= \log x^y + \left (\log x^{y'} + \frac{yx'}{x} \right )\epsilon \end{split}\]

Now we recover \(q = e^{\log q}\):

\[\begin{split}q &= \exp\left \{ \log x^y + \left (\log x^{y'} + \frac{yx'}{x} \right )\epsilon\right \} \\ &= x^y \cdot \exp \left \{\left (\log x^{y'} + \frac{yx'}{x} \right )\epsilon\right \} \end{split}\]

Now using \(e^{a\epsilon} = 1 + a\epsilon\), since \(\epsilon^2 = 0\):

\[q = x^y \left [1 + \left (y^\prime \log x + \frac{yx'}{x} \right )\epsilon \right ]\]

we would implement this in the __pow__ method of our Dual class.

Implementation#

Our Dual class holds 2 pieces of data: a value and a derivative.

We can implement most of the dual arithmetic by overloading operators. For some special functions, like sqrt(), we will have to write our own function.

class Dual:
    """Carry a quantity that can propagate its derivatve.

    We carry a scalar derivative, so that means we are always
    differentiating with respect to the same quantity.

    """

    def __init__(self, x, deriv=1):
        self.value = x
        self.deriv = deriv

    def __abs__(self):
        if self.value > 0:
            return Dual(self.value, self.deriv)
        if self.value < 0:
            return Dual(-self.value, -self.deriv)
        return Dual(abs(self.value), 0.0)

    def __lt__(self, other):
        if isinstance(other, Dual):
            return self.value < other.value
        return self.value < other

    def __gt__(self, other):
        if isinstance(other, Dual):
            return self.value > other.value
        return self.value > other

    def __eq__(self, other):
        if isinstance(other, Dual):
            return self.value == other.value
        return self.value == other

    def __add__(self, other):
        if isinstance(other, Dual):
            return Dual(self.value + other.value, self.deriv + other.deriv)
        return Dual(self.value + other, self.deriv)

    def __radd__(self, other):
        return self.__add__(other)

    def __sub__(self, other):
        if isinstance(other, Dual):
            return Dual(self.value - other.value, self.deriv - other.deriv)
        return Dual(self.value - other, self.deriv)

    def __rsub__(self, other):
        if isinstance(other, Dual):
            return other.__sub__(self)
        return Dual(other - self.value, -self.deriv)

    def __neg__(self):
        return Dual(-self.value, -self.deriv)

    def __mul__(self, other):
        if isinstance(other, Dual):
            # Product rule, using the original operand values.
            return Dual(self.value * other.value,
                        self.value * other.deriv + other.value * self.deriv)
        return Dual(self.value * other, self.deriv * other)

    def __rmul__(self, other):
        return self.__mul__(other)

    def __truediv__(self, other):
        if isinstance(other, Dual):
            # Quotient rule, using the original operand values.
            return Dual(self.value / other.value,
                        -self.value / other.value**2 * other.deriv + self.deriv / other.value)
        return Dual(self.value / other, self.deriv / other)

    def __rtruediv__(self, other):
        if isinstance(other, Dual):
            return other.__truediv__(self)
        return Dual(other / self.value, -other / self.value**2 * self.deriv)

    def __pow__(self, y):
        if isinstance(y, Dual):
            if self.value <= 0:
                raise ValueError("Dual exponents require a positive base")
            value = self.value**y.value
            deriv = value * (y.deriv * np.log(self.value)
                             + y.value * self.deriv / self.value)
            return Dual(value, deriv)

        if y == 0:
            return Dual(1.0, 0.0)
        return Dual(self.value**y,
                    y * self.value**(y - 1.0) * self.deriv)


def sqrt(x):
    """Compute sqrt(x)."""
    if isinstance(x, Dual):
        value = np.sqrt(x.value)
        return Dual(value, 0.5 * x.deriv / value)
    return np.sqrt(x)

Important

We need to be sure to call our version of sqrt() instead of the numpy version, so the Dual will properly be used.

Using our Dual#

Here’s our same equation of state as above, but we replace np.sqrt() with our new sqrt().

def pressure(rho, T):

    # ions are ideal gas
    p_ion = rho * k_B * T / (mu_I * m_u)

    # electrons can be ideal gas or degenerate -- we use an approximation to blend
    # the two regimes
    p_e_ideal = rho * k_B * T / (mu_e * m_u)
    p_e_deg = K * (rho / mu_e)**(5./3.)
    p_e = sqrt(p_e_ideal**2 + p_e_deg**2)

    return p_ion + p_e

Since we want to differentiate with respect to density, we make rho a Dual.

rho = Dual(1.e5)
T = 1.e7

Note

We do not initialize deriv explicitly here, so it takes on the value 1. This is basically saying that

\[\partial \rho /\partial \rho = 1\]

Now we can call our function as usual. All the math operations will use the Dual versions.

p = pressure(rho, T)

We can access p.value and p.deriv to get the results.

print(f"p = {p.value:10.5g}; ∂p/∂ρ |_T = {p.deriv:10.5g}")
p = 7.1246e+20; ∂p/∂ρ |_T = 1.1719e+16

These agree with our finite-difference approximation above.

Important

We cannot get the independent derivatives with respect to \(\rho\) and \(T\) by calling the function with two Dual representing different independent variables, since our Dual only holds a single derivative. So if we did:

rho = Dual(1.e5)
T = Dual(1.e7)

p = pressure(rho, T)

we would actually be computing

\[\left . \frac{\partial p}{\partial \rho} \right |_T + \left . \frac{\partial p}{\partial T} \right |_\rho\]

It is possible to create a Dual that holds an array of derivatives for multivariate differentiation.