Are there programs that never halt and have no non-termination proof?

Question

Like black holes in computer science. We can only know they exist but when we have one of them we will never know it's one of them.

Answer 1

There are indeed programs like this. To prove this, let's suppose to the contrary that for every machine that doesn't halt, there is a proof it doesn't halt.

These proofs are strings of finite length, so we can enumerate all proofs of length less than $s$ for some integer $s$.

We can then use this to solve the halting problem as follows: Given a Turing Machine $M$ and an input $x$, we use the following algorithm:

s := 0
while (True)
    test if machine M halts on input x in s steps
    look at all proofs of length s and see if they prove M doesn't halt on input x
    set s := s + 1

If $M$ halts on input $x$, then it halts in some finite number of steps $s$, so our algorithm terminates.

If $M$ doesn't halt on input $x$, then by our assumption, there's some proof length $s$ where there's a proof that $M$ doesn't halt. So in this case, our algorithm always terminates.

Thus, we have an algorithm deciding the Halting problem which always terminates. But we know this cannot exist, so our assumption that there's always a proof of non-halting must be false.

Answer 2

For a somewhat more concrete example, let us assume that the theory we are using for our proofs has the following (quite reasonable, IMO) features:

1. It is is consistent; that is, it cannot prove a contradiction.
2. Its set of axioms is recursively enumerable.
3. Its proofs can be written down as finite bitstrings.
4. The question of whether a given string encodes a well-formed and correct proof in it is algorithmically decidable in finite time.
5. It is expressive enough to admit a proof of Gödel's second incompleteness theorem, which says that it cannot prove its own consistency.

With those assumptions, the following program will never halt, but cannot be proved (within the scope of the theory we are using) not to halt:

let k := 0;
repeat:
    let k := k + 1;
    let s := binary expansion of k, excluding leading 1 bit;
while s does not encode a proof of a contradiction;
halt.

The key detail here is the first assumption above, namely that the theory we are using for our proofs is consistent. Obviously, we need to assume this, for our proofs to be worth anything, but Gödel's second incompleteness theorem says that, for any reasonably expressive and effectively axiomatized theory, we cannot actually prove this (except possibly in some other theory, whose consistency we then need to assume, etc. etc.).