The Enhanced MR test of FIPS 186-5 is passed an odd integer $w$ to be tested for primality, and returns one of 3 possible outcomes
- PROBABLY PRIME
- PROVABLY COMPOSITE WITH FACTOR, and a factor of $w$ in $\bigl[3,\lfloor w/3\rfloor\bigr]$
- PROVABLY COMPOSITE AND NOT A POWER OF A PRIME
PROVABLY can be removed from the statement of 2 or 3. Make sure to distinguish proBably from proVably: outcome 1 does not quite imply that $w$ is prime!
(does) the outcome of this enhanced MR test depends on the random witness chosen ?
Yes, except when the input $w$ is prime or small.
More precisely: possible outcomes according to the nature of $w$
| nature of $w$ being tested |
possible outcomes |
remark |
| Prime |
1 |
|
| Composite and a prime power |
1 or 2 |
1 is rare, or impossible for small enough $w$ |
| Composite not a prime power |
1 or 2 or 3 |
1 is rare, or impossible for small enough $w$ |
As observed in the question, for some pseudorandom values of the witness $d$ (generated at step 4.1 of the algorithm), the outcome is 2 for a composite $w$ that's not a power of a prime. The outcome can also be 2 when $w$ is a power of a prime. For example for $d=33$, the outcome is 2 with the factor $g=11$ returned both when $w=77$ and $w=121$, with only the later being a power of a prime.
If $w$ is random, outcome 2 is not uncommon. If $w$ is large and random among integers having no small prime factors (e.g. thanks to a sieving step when generating $w$), outcome 2 is rare. For products of two large distinct prime factors and low iteration count, outcome 2 is even rarer than outcome 1 is. There are values of $w$ making outcome 2 common for low iteration count, even though $w$ is a composite not a power of a prime. An example is the 2048-bit
0xfec7962ea307d2c8522415025ffa0dd7948c78d7c9daba1a87b9fd421d016598b5b8403306e3125a3ac8751353c2eab69ea490de4bc71a6fc688bacb0f8bae5c325d5427fc666ceff2bbba64ab2e803481343f2a4d94ae774f70411bcff299f9fc63234b4d22b19b641e6463f5db961603715c3ac59feed8309e7d28955c26d2a4c2929b48f3551fd9522ee7bc6177cee69e1839cae00b093f463f03fb79e099d5ccc057963b195b9cf04e14c933eb4522aa8c574f1e8e43714c3a5155932206609d1b987356383df3c6d0fa63be1fde017a42fd1ee36e6c64398912978db8b1c35d23244a986de6535a44d44b893961ba09f14d88458bc0f32424e4b85ea379
for primality test if I keep the iteration count to 64, it provides a bit strength of 2^-128 (that is the probability of generating a bad prime)
That probability is actually much smaller. Regardless of how $w$ was generated, for $i$ iterations, there's probability below $2^{-2i}$ that the outcome is PROBABLY PRIME when $w$ is not actually prime, that is probability below $2^{-256}$ for $i=64$. And if the candidate prime $w$ is large and generated randomly (including among integers having no prime factors below some limit), the probability is so much lower that for 1024-bit $w$, $i=7$ yields probability below $2^{-100}$ of generating a composite. See table B.1 or this, and for the math appendix C.1 and it's references.
Notice that for the purpose of generating primes, we can ignore the difference there is between outcomes 2 and 3. That's the case for all purposes in FIPS 186-5.
Update: @poncho gives a nice proof of the NOT A POWER OF A PRIME part of outcome 3.