Is there an algorithm to generate primes iteratively and not test for primality?

Question

I apologise if I'm not clear. I know that you can find all prime numbers between $1$ and $n$ by iteratively testing all numbers between $1$ and $n$ and recording those that pass a primality test. Is there a way to algorithmically find all the primes between $1$ and $n$ without testing each number for primality?

Answer 1

If you want to find all of the primes from 1 to $n$, the best way by far is a sieve. The sieve of Eratosthenes is easy to write and fast. How fast? The basic version takes time $O(n\log\log n)$, which mean that it spends $\log n\log\log n$ time for each prime. This is really quick. In practice this means that it's faster to generate primes than to read them from disk (though not as fast as reading them from memory). Testing each number for primality would take longer, in theory, and much longer in practice ($100\times$ longer at best). Let me put it this way: sieving doesn't take that much longer than just listing the primes that you find.

You might want to look at a range of numbers large enough that you can't store all the primes in memory. That's OK -- there are sieves that solve that problem. 99% of the time the answer is the same: the sieve of Eratosthenes can be segmented to require only $\sqrt n$ or so memory, so finding the primes up to $10^{15}$ would take 4 MB instead of 125 TB.

You mentioned a function to find the next prime after a given prime $p$. There's no good way to do this; at present the best algorithm would be to sieve the small primes from a chunk somewhat bigger than $\log p$, then test the remaining numbers until you find one which is prime.* But sieves have an answer here, too. If you hit the end of the sieve and decide you want to go further, you can—without extra effort!—if you're using a class of sieves called incremental sieves. Sorenson [1] has an example; other ones I've seen ([2], [3]) are impractical.

If you like the idea of a sieve, but $\log n\log\log n$ is too slow for you, there are more advanced sieves which take only $\log n$ work per prime, some of which can be combined with a wheel to reduce this to $\log n/\log\log n.$ The Atkin-Bernstein sieve is the usual example but there are others. (There's nothing special about that sieve in terms of speed, the paper describes a linear sieve and gives a general trick, related to what Tao calls the W-trick, for shaving a factor of $\log\log n$.) This is the fastest that any existing sieves can go, and it's not that far from the best possible. I mean, think about it -- writing a number $n$ in decimal takes $\Theta(\log n)$ time because it has that many digits, and so methods like this produce primes faster than you can write them down.

In practice highly-optimized versions of the sieve of Eratosthenes are state of the art. The best use a variant of Oliveira e Silva; primesive may be the fastest freely available sieve. Bernstein's PrimeGen is innovative but not presently competitive.

Speed isn't the only problem, though. If you want to find the primes up to a very large $n$ then you could run out of memory, or you could run out of memory because you're running on a limited machine. Even if you have plenty of memory you might not want to use it all—possibly because other programs need it, but more likely because main memory is too slow and you want to run the sieve in the L3 or even L2 cache. In that case the dissected sieve [4] is excellent, and [5] promises to be an interesting alternative. Both run in $O(n^{1/3+\varepsilon})$ space which should be all you'll ever need if you're running from 1 to $n$ (rather than large isolated intervals like $10^{25}$ to $10^{25}+10^9$). But just in case there are sieves which use even less space like the pseudosquare prime sieve [6], though at that point you pay a steep performance penalty. And of course you could always do a partial sieve by small primes followed by a Miller-Rabin test and a primality-proving routine, for another performance hit.

* If you were really committed to doing this, theory says you would want to sieve up to something like $\exp(\log^2p/\log\log p)$ and the sieve range could be, say, $p+2$ to $p+5\log p$ (so around 1% of the time you'd need to sieve out a second interval), then use a Miller-Rabin test on all numbers not sieved out until one passed; at that point you can use the primality-proving test of your choice, maybe BPSW if less than $2^{64}$ or ECPP otherwise.

References

[1] Jonathan P. Sorenson, Two compact incremental prime sieves, LMS Jour. Comp. Math. 18 (2015), pp. 675–683. arXiv:1503.02592 [cs.DS]

[2] S. Bengelloun, An incremental primal sieve, Acta Informatica 23:2 (1986), pp. 119–125.

[3] Paul Pritchard, Improved incremental prime number sieves, ANTS-I (May 1994), pp. 280–288.

[4] William F. Galway, Dissecting a sieve to cut its need for space. In Algorithmic Number Theory (Leiden, 2000), volume 1838 of Lecture Notes in Comput. Sci. (2000), pp. 297–312.

[5] Harald A. Helfgott, unpublished conference slides (2016).

[6] Jonathan P. Sorenson, The pseudosquares prime sieve