Taylor Series

Functions don’t always play nice with other functions. Nested non-linear functions can be particularly difficult to work with. Derivatives and polynomials are the Switzerland and Lichtenstein (respectively) of functions. They get along with pretty much all the other functions, and this explains the remarkable success of the Taylor expansion. The Taylor expansion is a way of approximating a finite part of any differentiable function as a really big polynomial. Let’s take the function $f(x)$, and try to approximate it at a value of $x=x_0+\epsilon$ , where $x_0$ is a constant and $\epsilon$  is a variable. We might logically start by saying

$$ f(x_0+\epsilon) \approx f(x_0), $$

but that really only works for very small values of $\epsilon$. As you get further away from $x_0$, you’ll need to somehow correct for how $f(x)$ changes as $x$  changes...enter, the derivative! The first derivative describes the slope of a function at any given point, so we could add a line to our approximation that matches that slope at that point:

$$ f(x_0+\epsilon) \approx f(x_0) + f'(x_0)\epsilon. $$

Now our approximation matches the target function’s rate of change immediately around that point. That’s great, but rates of change don’t stay the same forever...enter, the second derivative! Now it might be tempting to continue this pattern with $f(x_0)\epsilon^2$, but we need to be careful—the second derivative of that would be $2f(x_0)$. The second-order term should therefore be $\frac{1}{2}f''(x_0)\epsilon^2$ to adjust for that. Our approximation is now $$\label{eq:taylor:2} f(x_0+\epsilon) \approx f(x_0) + f'(x_0)\epsilon + \frac{1}{2}f''(x_0)\epsilon^2.$$ Let’s add a few more terms in the same way and watch the pattern emerge.

$$\begin{aligned} f(x_0+\epsilon) &\approx f(x_0) + f'(x_0)\epsilon+ \frac{1}{2}f(x_0)\epsilon^2+ \frac{1}{6}f'(x_0)\epsilon^3 + \frac{1}{24}f^{(4)}(x_0)\epsilon^4 \\   &\approx \frac{f(x_0)\epsilon^0}{0!} + \frac{f'(x_0)\epsilon^1}{1!} + \frac{f(x_0)\epsilon^2}{2} + \frac{f'(x_0)\epsilon^3}{3!} + \frac{f^{(4)}(x_0)\epsilon^4}{4!} +\cdots + \frac{f^{(n)}(x_0)\epsilon^n}{n!}\label{eq:taylor:n} \end{aligned}$$

This approximation gets better and better with more terms, and as $n$ goes to infinity, it will actually become an exact representation of the original function as long as the derivatives are well defined.

Now here’s where the magic happens. A factorial in the denominator will quickly dominate a monomial in the numerator, which means that the higher order terms will tend to be less and less important (especially if $\epsilon$ is small). Most functions can be approximated with just one or two non-zero terms! For a more detailed explanation of the Taylor expansion, as well as the expansions of some common functions, visit the Wikipedia article.