Approximate $|x|$ with a smooth function

Question

I am trying to get the derivative of $|x|$, and I want that derivative function, say $g(x)$, to be a function of x.

So it really needs the |x| to be smooth (ex. $x^2$); I am wondering what is the best way to approximate |x| with something smooth?

I propose $\sqrt{x^2 + \epsilon}$, where $\epsilon = 10^{-10}$, but there should be something better? Perhaps Taylor expansion?

Sorry for any confusion. I should add some additional information here:

I want to use $|x|$ as part of an object function $J(x)$ which I want to minimize. So it would be nice to approximate $|x|$ with some smooth function so that I can get the analytic form of the first-order derivative of $J(x)$.

Thanks a lot.

Answer 1

A bit late to the party, but you can smoothly approximate $f(x) = |x|$ by observing $\partial f/\partial x = \mbox{sgn}(x)$ for $x \neq 0$. Therefore approximating the $\mbox{sgn}(x)$ function by

$$ f(x) = 2\mbox{ sigmoid}(kx)-1$$

(the $k$ being a parameter that allows you to control the smoothness), we get

$$ \partial f/\partial x = 2\left (\frac{e^{kx}}{1+e^{kx}} \right ) - 1 $$ $$ \Rightarrow f(x) = \frac{2}{k}\log(1+e^{kx})-x-\frac{2}{k}\log(2)$$

where the constant term was chosen to ensure $f(0) = 0 $.

Included are plots of the function for $x \in [-5,5]$, where $k=1$ is red, $k=10$ is blue and $k=100$ is black.

Note that if you wanted to use the more smooth $k=1$ in the interval $[-5,5]$ it may be worth applying a further linear transformation i.e. ~$5f(x)/3.6$ to ensure the values at the edge of the interval are correct.

Answer 2

I have used your $\sqrt{x^2+\epsilon}$ function once before, when an application I was working on called for a curve with a tight radius of curvature near $x=0$. Whether it is best for your purpose might depend on your purpose.

(Side note: The derivative of $|x|$ does not exist at $x=0$. In physics, we sometimes cheat and write ${d\over dx}|x| = \rm{sgn}(x)$, the "sign" function that gives $-1$ when $x<0$, $0$ when $x=0$, and $+1$ when $x > 0$. But only when we know that the value at $x=0$ will not get us into trouble!)

Answer 3

Another solution would be the function $x \rightarrow x \frac{e^{kx}}{e^{kx} + e^{-kx}} -x\frac{e^{-kx}}{e^{kx} + e^{-kx}} $ with higher the $k$, closer you will be to the absolute value function. Furthermore it is equal to 0 in 0 wich is sometimes a desirable feature (compared to the solution with the root square stated above). However you lose the convexity.

Answer 4

From this question I did and the answer given by @LPZ you could have the following approximations for the Absolute value:

1. $$A(x):\approx \lim_{\varepsilon\to\infty}\begin{cases}\dfrac{2}{\varepsilon},\quad x=0\\ x\coth\left(\dfrac{\varepsilon x}{2}\right),\,x\neq 0\end{cases}$$
2. $$B(x):\approx \lim_{\varepsilon\to\infty} \dfrac{\ln\left(2\cosh\left(\varepsilon x\right)\right)}{\varepsilon}$$

You could see how they approximate $|x|$ as $\varepsilon$ increases in Desmos.

As example, $B'(x)=\tanh(\varepsilon x)$ gives you a smooth approximation for $\text{abs}'(x)$ if you allow it to be defined as $\text{sgn}(x)$.

comment added later: notice both functions aren't equivalent to the hyperbola solution $\sqrt{\varepsilon+x^2}$ given by @JasonZimba for any $\varepsilon$.

Answer 5

As you and others have mentioned, functions of the form $\sqrt{x^2 + 4\mu^2}$ can be a good approximation to $\left|x\right|$ and are standard in many optimization applications where the smoothing function meets certain bounding requirements. See, for example, the paper Optimality Conditions and a Smoothing Trust Region Newton Method for Non-Lipschitz Optimization where that function is used to develop a smoothing optimization algorithm.

Another smooth approximation that has been proposed in Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches is

$$ \begin{equation} \mid x\mid \ \approx \frac{1}{\alpha} \left[ \log\left(1 + \exp(-\alpha x)\right) + \log\left(1 + \exp(\alpha x) \right)\right] \end{equation} $$ where $\alpha$ is taken to be a large positive number.

Answer 6

Notice that $|x| = \max\{x, -x\}$. There are a few notions of Smooth Maximum and Boltzman operator version of smooth maximum is illustrated on the page as a means of approimating $|x|$. One of the simplest not mentioned on the preceding Wikipedia page is is the $\log$-semiring where in essence one can choose a large base $b$ and use:

$$\log_{b}(b^{x} + b^{-x})$$

as

$$\lim_{b \to \infty} \log_{b}(b^{x} + b^{-x}) = \max\{x, -x\} = |x|$$

Answer 7

I had a similar problem and here is the solution I am using the following for x in range (-pi, pi) using fourier, it's also infinitely differentiable everywhere: $$0.04 + (-.1 + \pi - 8\cos(x)/\pi - 8\cos(3x)/9\pi -8\cos(5x)/25\pi)*0.52 - 8\cos(7x)/45\pi$$

And here is the image of the plot and y = abs(x) for comparison: Plot Picture

It also has a nice property for few cases where abs(a - b) needs to be expanded, which can be done so via the cosine rule. This becomes beneficial especially when numerically solving coupled ODEs where the $\dot Y$ depends on $\sum_{i, j}\mid(y_{j} - y_{i})\mid$.

Expanding the cosine rule here allows you to distribute the summations and precompute the sums.