What are some cases where we can prove that AGI is always going to be wrong atleast some of the times? One case i can think of is any fact retrieval. For example, just retrieving the date of births of public figures, we know AGI is going to make mistake just because the way it generates the answer probablistically and the answer is sharply defined. Even if it becomes good at maths, date of birth is a rote memorization problem and any probablistic method is bound to fail at times. Now we can teach it to actually go to internet and double check its answer for such cases but it is not possibly to so preemptively, so its always going to be unreliable, even if with small probablity. Maybe when its error rate becomes lesser than that of a normal human, we can call it better than human intelligence as a concession.
What are some other hard constraints?
