How Well Do Logarithms Preserve Properties Of A Function?

Question

This is a question I have always had, relating to probability and statistics.

In many applications (e.g. estimating the parameters of a probability distribution function), we almost always end up trying to optimize the "log likelihood" instead of just the "likelihood".

From a computational standpoint, I have heard that this is much easier - for example, diffrentiating the log likelihood can remove exponent terms and thus make the optimization process simpler.

From a mathematical standpoint, we are often told (without explanation) that optimizating the log likelihood function is equivalent to maximizing the original likelihood function - for example, the stationary points (i.e. where the derivatives are 0) on the log likelihood function are apparently equivalent to the stationary points on the original likelihood function. Therefore, optimizing the log likelihood function or the original likelihood function will result in identical parameter estimates.

My question relates to the mathematics of this phenomenon : Does the logarithm of a function always preserve the stationary points of the original function - and if so, why does this happen?

As a reference, I found the following quote (Why we consider log likelihood instead of Likelihood in Gaussian Distribution):

"Because the logarithm is monotonically increasing function of its argument, maximization of the log of a function is equivalent to maximization of the function itself."

Thus - how do I know that the above is true?

I will assume that "a logarithm is montonically increasing function of its argument" is true by definition - can we mathematically prove that :

1. Maximizing the log of a function is ALWAYS equivalent to maximizing the function itself?

2. Given any montonically increasing function - does maximizing a function and any montonically increasing transformation of this original function ALWAYS results in identical results?

Thanks!

Answer 1

It is not true that the logarithm is monotonically increasing “by definition”. It's a property that has to be proved. If, for instance, you define $\log x$ as $\int_1^x\frac{\mathrm dt}t$, then, if $0<x<y$,\begin{align}\log(y)-\log(x)&=\int_1^y\frac{\mathrm dt}t-\int_1^x\frac{\mathrm dt}t\\&=\int_x^y\frac{\mathrm dt}t\\&>0,\end{align}and therefore $\log(x)<\log(y)$.

If $F\colon\Bbb R_+\longrightarrow\Bbb R$ is monotonically increasing and if $f\colon A\longrightarrow\Bbb R_+$ is a function (with $A\subset\Bbb R$) which attains its maximum at some $a_0\in A$, then, yes, $F\circ f$ attains its maximum at $a_0$ too. That's so because, if $a\in A$, then $f(a)\leqslant f(a_0)$, and therefore, since $F$ is monotonically increasing, we also have $F\bigl(f(a)\bigr)\leqslant F\bigl(f(a_0)\bigr)$. On the other hand, if $F\circ f$ attains its maximum at $a_0$, then for every $a\in A$, we have to have $f(a)\leqslant a_0$; otherwise, there would be some $a\in A$ such that $f(a)>f(a_0)$, in which case $F\bigl(f(a)\bigr)>F\bigl(f(a_0)\bigr)$.

Answer 2

Jose Carlos Santos perfectly answered about the monotonicity of the logarithm. I will try to answer the other question.

Assume that $g$ is an increasing function with domain $D$ included in $\mathbb R$ and $f$ any function. For example, $g$ could be any logarithm with a base $a>1$. Here is a list of properties that are preserved by $g$. Proving those facts is a good exercise. They follow from the definitions and the fact that $g$ is one-to-one, hence has an inverse (with domain the range of $g$) and $g^{-1}$ is also increasing.

The list:

A. $f$ is increasing (resp. decreasing) if and only if $g\circ f$ is increasing (resp. decreasing).

B. $f$ has a absolute maximum at $a$ (resp. local minimum at $a$) if and only if $g\circ f$ has a absolute maximum at $a$ (resp. local minimum at $a$)

C. $f$ has a local maximum at $a$ (resp. local minimum at $a$) if and only if $g\circ f$ has a local maximum at $a$ (resp. local minimum at $a$)

I think that we cannot say anything about the concavity of $f$ from the concavity of $g\circ f$.

A last one:

If $f$ is even, then $g\circ f$ is also even.

Note that if $f$ is odd, then $g\circ f$ may or may not be even.