set of words w such that M halts on w is decidable

Question

I need to prove that the language following language is not turing-recognizable: $$\text{dec-haltTM} = \{ \langle M\rangle: \text{$M$ is a TM and the set of words that M halts on is decidable}\}$$

I had the following reduction from $\overline{A_{TM}}$ in mind:

"On input $\langle M, w\rangle$ such that $M$ is a TM and $w$ is a string:

1. build a TM $M'$ on input that on input $x$: 1.1 simulate the run of $M$ on $w$. If the simulated run is accepting, accept $\langle M, w\rangle$. 1.2 else enter a loop.
2. return $\langle M'\rangle$."

but I'm not sure I can return the output in line 1.1. I tried to create $M'$ such that if $\langle M,w \rangle$ is in $\overline{A_{TM}}$ then $M'$ doesn't halt on any input thus the language is $\emptyset$ regular and decidable. If $\langle M,w\rangle$ is in $A_{TM}$, then the set of words $M'$ halts on is $A_{TM}$, thus not decidable.

Answer 1

You cannot let $M'$ recognize $A_{TM}$ by letting it "accept $\langle M, w\rangle$". You need to define the behaviour of $M'$ on $x$, which you completely ignored. What you wanted to achieve can be implemented by letting $M'$ recognize an undecidable language $L$ when $M$ accepts $w$, yet does not halt on any input in $\overline{L}$. It is not hard to define the reduction so that the language $L$ satisfies $L= A_{TM}$, as you wanted. Specifically:

Let $T$ be a fixed TM with $L(T) = A_{TM}$. A reduction from $\overline{A_{TM}}$ to decHaltTM operates as follows. Given input $\langle M, w \rangle$ for the reduction, the reduction outputs $\langle M'\rangle$ that operates as follows. Given input $x$ for $M'$, $M'$ simulates the run of $M$ on $w$. If the simulated run is rejecting, then $M'$ enters a loop. If the simulated run is accepting, then $M'$ simulates the run of $T$ on $x$, and answers the same when $T$ accepts $x$, and otherwise, if $T$ rejects $x$, $M'$ enters a loop.

Regarding correctness, it is not hard to see that if $M$ accepts $w$, then the language of $M'$ is identical to the language of $T$, and thus $L(M') = L(T) = A_{TM}$. Then, as $M'$ is defined so that it does not halt on any input in $\overline{A_{TM}}$, then the set of words it halts on equals its language, in particular, is not decidable. Hence $\langle M'\rangle \notin \text{dec-haltTM}$.

Coversely, if $M$ does not accept $w$, then $M'$ does not halt on any input $x$, and thus the set of inputs that $M'$ halts on is empty, in particular decidable. Hence $\langle M' \rangle \in \text{dec-haltTM}$.