Auto-differentiation#
Quite often we need the derivative of a function we are working with. If our function is very complex, it may not be possible to differentiate it analytically. This is where automatic differentiation or “autodiff” can help.
Consider the following function, which is a (crude) approximation to the equation of state for a degenerate gas + ideal gas nuclei (we used this in Application: One-zone X-ray burst):
import numpy as np
First we’ll define the necessary fundamental constants:
k_B = 1.38e-16 # erg / K
h = 6.67e-27 # erg s
m_e = 9.11e-28 # g
m_u = 1.66e-24 # g
# electron degeneracy
K = h**2 /(20 * m_e) * (3.0 / np.pi)**(2./3.) * m_u**(-5./3.)
# we'll use pure He for the compoosition
mu_I = 4.0
mu_e = 2.0
Now our function:
def pressure(rho, T):
# ions are ideal gas
p_ion = rho * k_B * T / (mu_I * m_u)
# electrons can be ideal gas or degenerate -- we use an approximation to blend
# the two regimes
p_e_ideal = rho * k_B * T / (mu_e * m_u)
p_e_deg = K * (rho / mu_e)**(5./3.)
p_e = np.sqrt(p_e_ideal**2 + p_e_deg**2)
return p_ion + p_e
Given a \(\rho, T\), we can get the total pressure. But what if we wanted \(\partial p/\partial \rho |_T\) or \(\partial p/\partial T|_\rho\)?
Finite-difference method#
We could use a finite-difference, like we did in Differentiation.
First, let’s define the thermodynamic state:
rho = 1.e5
T = 1.e7
Now we can get the pressure
p = pressure(rho, T)
We can use a centered-difference to get the derivative:
eps = 1.e-8
drho = rho * eps
dpdrho = (pressure(rho+drho, T) - pressure(rho-drho, T)) / (2.0 * drho)
print(f"p = {p:10.5g}; ∂p/∂ρ |_T = {dpdrho:10.5g}")
p = 7.1246e+20; ∂p/∂ρ |_T = 1.1719e+16
Note
For complex functions, a finite-difference can be slow. It can also suffer from accumulation of round-off error and result in an inaccurate derivative.
Introduction to automatic differentiation#
The basic idea of autodiff is to define a new datatype that carries both a value and a derivative and understands how mathematical operations propagate the derivative. We call this new type dual numbers or a “dual”.
When we work with a dual, it essentially just uses the chain rule automatically for us.
Note
We will implement “forward mode” autodiff, which works well when we are only differentiating with respect to a few variables.
Basic arithmetic operators#
For a quantity \(x\), we replace it with \(x + x'\epsilon\), where \(\epsilon\) is a special mathematical quantity (a formal infinitesimal) with the property \(\epsilon^2 = 0\). The coefficient \(x'\) carries the derivative with respect to the independent variable.
With this definition, we have:
addition: \((x + x'\epsilon) + (y + y'\epsilon) = x + y + (x' + y')\epsilon\)
subtraction: \((x + x'\epsilon) - (y + y'\epsilon) = x - y + (x' - y')\epsilon\)
multiplication: \((x + x'\epsilon) \cdot (y + y'\epsilon) = xy + (xy' + yx')\epsilon\)
division:
\[\frac{x + x'\epsilon}{y + y'\epsilon} = \frac{\frac{x}{y} + \frac{x'\epsilon}{y}}{1 + \frac{y'\epsilon}{y}} = \left (\frac{x}{y} + \frac{x'\epsilon}{y} \right ) \left (1 - \frac{y'\epsilon}{y} \right ) = \frac{x}{y} + \left (\frac{x'y - xy'}{y^2} \right )\epsilon\]
We see that these just give the expression from calculus that we are familiar with.
Complex mathematical operations#
What about complex functions, like \(x^y\)? We can use the same ideas.
Start with:
It’s easiest to take the log first
Important
This assumes that \(x > 0\)
Now we recover \(q = e^{\log q}\):
Now using \(e^{a\epsilon} = 1 + a\epsilon\), since \(\epsilon^2 = 0\):
we would implement this in the __pow__ method of our Dual class.
Implementation#
Our Dual class holds 2 pieces of data: a value and a derivative.
We can implement most of the dual arithmetic by overloading operators. For some special functions,
like sqrt(), we will have to write our own function.
class Dual:
"""Carry a quantity that can propagate its derivatve.
We carry a scalar derivative, so that means we are always
differentiating with respect to the same quantity.
"""
def __init__(self, x, deriv=1):
self.value = x
self.deriv = deriv
def __abs__(self):
if self.value > 0:
return Dual(self.value, self.deriv)
if self.value < 0:
return Dual(-self.value, -self.deriv)
return Dual(abs(self.value), 0.0)
def __lt__(self, other):
if isinstance(other, Dual):
return self.value < other.value
return self.value < other
def __gt__(self, other):
if isinstance(other, Dual):
return self.value > other.value
return self.value > other
def __eq__(self, other):
if isinstance(other, Dual):
return self.value == other.value
return self.value == other
def __add__(self, other):
if isinstance(other, Dual):
return Dual(self.value + other.value, self.deriv + other.deriv)
return Dual(self.value + other, self.deriv)
def __radd__(self, other):
return self.__add__(other)
def __sub__(self, other):
if isinstance(other, Dual):
return Dual(self.value - other.value, self.deriv - other.deriv)
return Dual(self.value - other, self.deriv)
def __rsub__(self, other):
if isinstance(other, Dual):
return other.__sub__(self)
return Dual(other - self.value, -self.deriv)
def __neg__(self):
return Dual(-self.value, -self.deriv)
def __mul__(self, other):
if isinstance(other, Dual):
# Product rule, using the original operand values.
return Dual(self.value * other.value,
self.value * other.deriv + other.value * self.deriv)
return Dual(self.value * other, self.deriv * other)
def __rmul__(self, other):
return self.__mul__(other)
def __truediv__(self, other):
if isinstance(other, Dual):
# Quotient rule, using the original operand values.
return Dual(self.value / other.value,
-self.value / other.value**2 * other.deriv + self.deriv / other.value)
return Dual(self.value / other, self.deriv / other)
def __rtruediv__(self, other):
if isinstance(other, Dual):
return other.__truediv__(self)
return Dual(other / self.value, -other / self.value**2 * self.deriv)
def __pow__(self, y):
if isinstance(y, Dual):
if self.value <= 0:
raise ValueError("Dual exponents require a positive base")
value = self.value**y.value
deriv = value * (y.deriv * np.log(self.value)
+ y.value * self.deriv / self.value)
return Dual(value, deriv)
if y == 0:
return Dual(1.0, 0.0)
return Dual(self.value**y,
y * self.value**(y - 1.0) * self.deriv)
def sqrt(x):
"""Compute sqrt(x)."""
if isinstance(x, Dual):
value = np.sqrt(x.value)
return Dual(value, 0.5 * x.deriv / value)
return np.sqrt(x)
Important
We need to be sure to call our version of sqrt() instead of the numpy version, so the Dual
will properly be used.
Using our Dual#
Here’s our same equation of state as above, but we replace np.sqrt() with our new sqrt().
def pressure(rho, T):
# ions are ideal gas
p_ion = rho * k_B * T / (mu_I * m_u)
# electrons can be ideal gas or degenerate -- we use an approximation to blend
# the two regimes
p_e_ideal = rho * k_B * T / (mu_e * m_u)
p_e_deg = K * (rho / mu_e)**(5./3.)
p_e = sqrt(p_e_ideal**2 + p_e_deg**2)
return p_ion + p_e
Since we want to differentiate with respect to density, we make rho a Dual.
rho = Dual(1.e5)
T = 1.e7
Note
We do not initialize deriv explicitly here, so it takes on the value 1. This is
basically saying that
Now we can call our function as usual. All the math operations will use the Dual versions.
p = pressure(rho, T)
We can access p.value and p.deriv to get the results.
print(f"p = {p.value:10.5g}; ∂p/∂ρ |_T = {p.deriv:10.5g}")
p = 7.1246e+20; ∂p/∂ρ |_T = 1.1719e+16
These agree with our finite-difference approximation above.
Important
We cannot get the independent derivatives with respect to \(\rho\) and \(T\) by calling the function with two Dual representing different independent variables, since
our Dual only holds a single derivative. So if we did:
rho = Dual(1.e5)
T = Dual(1.e7)
p = pressure(rho, T)
we would actually be computing
It is possible to create a Dual that holds an array of derivatives for multivariate differentiation.