For questions about the precision of machine calculations. This includes questions about how precise the result is, why this impreciseness exists and how to obtain more precise results.
Questions tagged [machine-precision]
91 questions
26
votes
3 answers
Why do different calculators disagree on $\cos(452175521116192774 )$?
I want to calculate cosine of 452175521116192774 radians (it is around $4.52\cdot10^{17}$)
Here is what different calculators say:
Wolframalpha
Desmos
Geogebra
Python 3.9 (standard math module)
Python 3.9 (mpmath library)
Obviously there is…
musava_ribica
- 405
14
votes
1 answer
Why does this desmos plot of the integral of $\sqrt{1+e^x}$ have these discontinuities?
I computed the integral of $\sqrt{1+e^x}$ by hand and got
$$2\sqrt{1+e^x} + \ln\left(\sqrt{1+e^x} - 1\right) - \ln\left(\sqrt{1+e^x} + 1\right) + C,$$
or
$$2\sqrt{1+e^x} + \ln\left(\frac{\left(\sqrt{1+e^x} - 1\right)^2}{e^x}\right) + C.$$
But when I…
Zhiyuan Liu
- 371
8
votes
2 answers
Reduce precision of fraction
Say I have a reduced fraction where the numerator and denominator can only be integers:
$$ \frac{1071283}{28187739} $$
and I want to reduce it more, accepting the lose of precision.
I could just remove an equal numbers of integers from the right:
$$…
Mobz
- 191
7
votes
4 answers
Multiply a very large integer by a very small probability
I am calculating the probability of rejecting a one-tailed hypothesis test for inequality of two proportions $p_1$ and $p_2$ (i.e., power) using $H_o: p_2 \leq p_1$ vs. the alternative $H_a: p_2 > p_1$
using the exact…
wjktrs
- 207
4
votes
2 answers
Numerically find cubic polynomial roots where coefficients widely vary in magnitude
Consider the following polynomial:
$$
p(x) = x^3 + (C_b+K_a)x^2 - (C_aK_a + K_w)x - K_aK_w
$$
Where:
$x, C_a, C_b$ are concentrations, positive real numbers typically within $[10^{-7};1]$. The unknown $x$ is the proton concentration and $C_a, C_b$…
jlandercy
- 512
- 6
- 19
4
votes
1 answer
Machine Epsilon meaning
Say we have the floating-point system $(2,3,-1,2)$ and we want to find machine epsilon. According to my textbook, this can be found as $\epsilon_m=\beta^{1-t} = 2^{1-3}=0.25$. However, my textbook also says that $\epsilon_m$ represents the distance…
Bee
- 285
4
votes
2 answers
How many digits of accuracy will an answer have?
I was doing a project Euler problem where I needed to find several Fibonacci numbers, but their index was so large that I could not use the typical recursive method. Instead, I used Binet's rule:
$$ F_n = \frac{\phi^n - \psi^n}{\sqrt{5}}$$
But since…
Ryan
- 1,220
3
votes
2 answers
Weird discrepancy in the calculation of an integral.
Integral in question, and its answer
The integral $$\int_{0}^{1000} e^{\left\{ x \right\}}\, \operatorname{d}x$$
where $\left\{\cdot \right\}$ is the fractional part function, evaluates to $1716.02$ in Desmos, which should THEORETICALLY be equal to…
sachyrus
- 43
3
votes
1 answer
Numerically stable way to compute ugly double fraction
I am looking for a numerically stable version of this (ugly) equation
$$
s^2=\frac{1}{\frac{1}{\beta_1}+\frac{1}{\beta_2}W}
$$
where
$$
\beta_1 = c_1-c_2m+(m-c_2)b\\
\beta_2 = \frac{1}{2}\left((a-m)^2-(b-m)^2\right)
$$
All symbols are real numbers…
mto_19
- 292
3
votes
2 answers
Why plot of a sinusoid with a large phase appears like a staircase?
I plotted the following in MATLAB and Desmos:
y = cos(x + 6998666554443343) (1)
The plot is shown here:
Plot of (1)
This staircase behaviour seems to appear with any large number. The following plot with an extra digit in its phase is another…
Manvendra Sharma
- 33
- 4
3
votes
0 answers
Limit of Euler's number failing due to precision errors - A surprising case. Why does it happen?
It is a known fact that floating point precision errors are bound to happen when one forces a computer to deal with very large or very small numbers, especially when both things are done at the same time. I was playing around with similar functions…
Danilo Guimarães
- 101
3
votes
1 answer
Avoiding catastrophic cancellation in $1-\operatorname{sinc}x$
When I try to calculate the function $f(x)=1-\operatorname{sinc}x$
for small values of $x$
I get large relative errors
due to catastrophic cancellation.
I want an accurate way to calculate $f(x)$
without using a series expansion
or an iterative…
Ken Whatmough
- 243
3
votes
1 answer
Why are SDP solvers inherently inaccurate?
It's common to read in textbooks that semidefinite programming solvers are inherently inaccurate.
Are the authors referring to the general machine inaccuracy (things like $10^{-16}=0$) or a special case around SDPs only?
If it is a special case,…
Koto
- 949
3
votes
1 answer
Accurate computation of square of complex values in Matlab: abs(X).^2 or X.*conj(X)?
I have found that in a dataset I am using I get different results depending on how I tell Matlab to compute the power. If I submit the command Xpow=abs(X).^2; vs. Xpow=X.*conj(X);, I get different results. The latter is much faster to compute, but…
NWalk
- 33
3
votes
1 answer
Rounding unit vs Machine precision
I'm not sure if this question should be asked here...
For a general floating point system defined using the tuple $(\beta, t, L, U)$, where $\beta$ is the base, $t$ is the number of bits in the mantissa, $L$ is the lower bound for the exponent and…
user168764