1

I am reading An Introduction to the Conjugate Gradient Method Without the Agonizing Pain. On page 20, the author wrote

The convergence results for Steepest Descent are

$$ \lVert e_{(i)} \rVert_A \le \left(\frac{\kappa-1}{\kappa+1}\right)^i \lVert e_{(0)} \rVert_A $$

On page 37, the author points out that

Suppose we wish to perform enough iterations to reduce the norm of the error by a factor of $\epsilon$; that is, $\lVert e_{(i)} \rVert_A \le \epsilon \lVert e_{(0)} \rVert_A$. Equation 28 can be used to show that the maximum number of iterations required to achieve this bound using Steepest Descent is $$ i \le \left\lceil\frac{1}{2} \kappa \ln\left(\frac{1}{\epsilon}\right) \right\rceil $$

This seems trivial but I cannot prove it. Here is my attempt:

$$ \begin{align} \lVert e_{(i)} \rVert_A \le& \epsilon \lVert e_{(0)} \rVert_A \\ \left(\frac{\kappa-1}{\kappa+1}\right)^i \le& \epsilon \\ i \ln\left(\frac{\kappa-1}{\kappa+1}\right) \le& \ln \epsilon \\ i \ln\left(\frac{\kappa+1}{\kappa-1}\right) \ge& \ln \left(\frac{1}{\epsilon}\right) \\ i \ge& \ln\left(\frac{\kappa+1}{\kappa-1}\right)^{-1} \ln \left(\frac{1}{\epsilon}\right) \end{align} $$

How do I proceed?

nalzok
  • 866

1 Answers1

1

It is a known lower bound for logarithm: $$\ln(1 + x) \ge \frac{2x}{2 + x}, \forall x \ge 0. \tag{1}$$ Simply letting $x = \frac{2}{k - 1}$ in (1), we have $$\ln\frac{k+1}{k-1} \ge \frac{2}{k}.$$

Then we have $$\left(\ln\frac{k+1}{k-1}\right)^{-1}\ln\frac{1}{\epsilon} \le \frac{k}{2}\ln\frac{1}{\epsilon}.$$

River Li
  • 49,125
  • Thanks for pointing the lower bound out! However, I believe there is some confusion between $\le$ and $\ge$. The paper says $i \le ...$, which is very likely a typo because $\lVert e_{(i)} \rVert_A \le \epsilon \lVert e_{(0)} \rVert_A$ clearly requires a lower bound on $i$, not an upper bound. As a result, what we need is a lower bound on $\left(\ln\frac{k+1}{k-1}\right)^{-1}$, or equivalently an upper bound on $\ln\frac{k+1}{k-1}$. – nalzok Jun 10 '23 at 01:59
  • 1
    @nalzok Yes, it is confusing. We require $i \ge \left(\ln\frac{k+1}{k-1}\right)^{-1}\ln\frac{1}{\epsilon}$. What I got is that the optimal $i$ does not exceed $\lceil \frac{k}{2}\ln\frac{1}{\epsilon} \rceil$. – River Li Jun 10 '23 at 02:25
  • 1
    This is true, and in the author's defense, when $\kappa \to \infty$ we have $\left(\ln\frac{\kappa+1}{\kappa-1}\right)^{-1} \approx \frac{\kappa}{2}.$ – nalzok Jun 10 '23 at 03:49