21

I was studying Numerical Analysis by K. Mukherjee; there he discussed Loss of Significant Figures by Subtraction, as followed:

In the subtraction of two approximate numbers, a serious type of error may be present when the numbers are nearly equal. ...

He showed how the number of significant figures got decreased after subtraction.

In order to dig more, I googled and came across the term catastrophic cancellation.

I did try to read the wiki article but it was full of terms especially floating point arithmetic which I am really not acquainted it; I did try to read the latter's wiki article but couldn't comprehend the definition:

The term floating point refers to the fact that a number's radix point (decimal point, or, more commonly in computers, binary point) can "float"; that is, it can be placed anywhere relative to the significant digits of the number.

Radix point can float? Unfortunately, I failed to visualise that.

However, my main question, even if there is an error due to the loss of the significant figures, why is this error "catastrophic"?

I found in a decade old page that

The relative error here is infinite.

But how could the loss of significant figures led to an infinite magnitude of relative error?

Could anyone please explain the reason behind the term "catastrophic" keeping in mind that I'm not acquainted with floating point arithmetic?

Rodrigo de Azevedo
  • 23,223
  • 6
    You are going to *have* to understand floating point arithmetic before any of these answers will make any sense, or indeed, before reading any book entitled Numerical Analysis. It isn't that hard, though. – Nate Eldredge Sep 09 '16 at 15:29
  • 1
    It is called catastrophic, since it leads to a serious error in the computations. So if you calculate for example a trajectory describe by a system of ordinary differential equations, the errors sum up and will lead to a very different (wrong) trajectory. But I support Nate Eldredge's comment. – TheGeekGreek Sep 09 '16 at 15:31
  • 7
    @NateEldredge Considering how many professional programmers don't understand floating point arithmetic, "It isn't that hard" may be a tad optimistic. – Daniel Fischer Sep 09 '16 at 15:32
  • Well, then @NateEldredge, thanks for the advise; the book I'm following doesn't even mention that; that's why. –  Sep 09 '16 at 15:32

1 Answers1

39

The root of the evil is that calculators work with a finite number of digits (well, we can't afford machines with infinite resources yet). When you want to represent real-life numbers, some can be huge, some can be tiny, and this raises an issue.

Floating-point:

If the decimal point is fixed, you can' represent them all, there can be so-called overflow or underflow; and for intermediate orders of magnitude, you lose significant digits.

For example, using the $5.5$ fixed-point representation, $127500.$ and $0.0000096695$ cannot be represented, and $\pi=3.14159$ just uses six significant digits, four positions are wasted.

The floating-point representation solves these problems by specifying the position of the decimal point, as in the scientific notation:

$127500.=1.275000000\cdot10^5, 0.0000096695=9.669500000\cdot10^{-6}, \pi=3.141592654\cdot10^0$.

Catastrophic cancellation:

(This concept is actually unrelated to floating-point.)

Assume you want to play with the approximation $\pi\approx355/113=3.14159292\cdots$.

If you want to take the sum with $\pi$, all goes well.

$$\frac{3.14159265+3.14159292}2=3.14159278.$$

But now if you want to take the difference,

$$\frac{3.14159265-3.14159292}2=-0.000000135,$$

only three significant decimals remain. This is a catastrophy, because you started with nine figures and end-up with just three, and this is irreversible ! (Floating-point will not come to the rescue, $-0.000000135=-1.35\cdot10^{-7}$ still has three significant digits.)

This phenomenon makes some problems very difficult to solve numerically.

In some cases, catastrophic cancellation can be avoided. For example, when solving the quadratic equation

$$x^2-1000.001x+1=0,$$

the classical formula says

$$x=\frac{1000.001\pm\sqrt{1000.001^2-4}}2=\frac{100.001\pm999.999}2.$$

Assuming you can only compute with five significant digits, using floating-point, you obtain

$$x=\frac{1000.00\pm999.99}2=999.95\text{ and }0.005,$$ where the second root is very very inaccurate.

You can get a much better estimate by using the fact that the product of the roots is $1$, and the second root evaluates to

$$\frac1{999.95}=0.0010005$$

  • Thanks, for the answer. I wanted to know why in that linked page in my post above, it was said, the relative error is infinite; could you please shed some light on this? –  Sep 09 '16 at 15:58
  • 1
    Also, unless you intentionally used \cdot, you could use \times to render $-2.7 \times 10^{-7};.$ –  Sep 09 '16 at 16:01
  • 1
    Or else never mind :) –  Sep 09 '16 at 16:02
  • @MAFIA36790: a matter of taste, I find the $\times$ a little intrusive. Anyway I admit that there can be confusion between $.$ and $\cdot$ –  Sep 09 '16 at 16:07
  • 1
    @MAFIA36790: my numerical shows a relative error of $400%$. You can find an example with $\infty%$. –  Sep 09 '16 at 16:09
  • Accepted; thanks for your efforts; however wouldn't the second root be $1/999.95;?$ another word, you have used 5.5 fixed point representation; could you provide some link where I could get a picture of them; I'm really not acquainted with that. Thanks again :) –  Sep 09 '16 at 17:01
  • @MAFIA36790: you are right. Fixed. –  Sep 09 '16 at 17:32
  • With only five significant digits, $999.999$ (or more specifically, $\sqrt{10^6 - 4}$) would round to $1000.0,$ and so would $1000.001$, which I think leaves you with one correct root and one root that's wrong by $100%$ (or $\infty%$, depending on whether you measure the error relative to the correct answer or the incorrect answer). – David K Aug 25 '19 at 20:53
  • @YvesDaoust Does catastrophic cancellation always lead to high relative error (sorry to awaken an old post) ? – M B Mar 13 '20 at 00:08
  • 2
    @JulianAngussmith: of course. –  Mar 13 '20 at 06:36