Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a differentiable approximation to the maximum function

$\{x_{1},\ldots ,x_{n}\}\mapsto \max\{x_{1},\ldots ,x_{n}\},\,$

and the concept of smooth minimum is similarly defined.

For large positive values of the parameter $\alpha >0$, the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

${\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n})={\frac {\sum _{{i=1}}^{n}x_{i}e^{{\alpha x_{i}}}}{\sum _{{i=1}}^{n}e^{{\alpha x_{i}}}}}$

${\mathcal {S}}_{\alpha }$ has the following properties:

1. ${\mathcal {S}}_{\alpha }\to \max$ as $\alpha \to \infty$
2. ${\mathcal {S}}_{0}$ is the average of its inputs
3. ${\mathcal {S}}_{\alpha }\to \min$ as $\alpha \to -\infty$

The gradient of ${\mathcal {S}}_{{\alpha }}$ is closely related to softmax and is given by

$\nabla _{{x_{i}}}{\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n})={\frac {e^{{\alpha x_{i}}}}{\sum _{{j=1}}^{n}e^{{\alpha x_{j}}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n}))].$

This makes the softmax function useful for optimization techniques that use gradient descent.

Another formulation is:

${\displaystyle g(x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n}))}$

References