Proof that {$a^m b^n$ | m!=n} is not regular

Question

I know that the language $\{a^m b^n | n\neq m\}$ satisfies the pumping lemma, but it's still not regular (I have to count the # of a's and b's). How can I formally prove it?

Answer 1

Suppose $L = \{ a^m b^n : m \neq n \}$ were regular. Then $a^*b^* \setminus L = \{ a^n b^n \}$ would also be regular.

There is a variant of the pumping lemma in which you mark certain symbols of your choice (at least as many as the pumping constant $p$), and then the pumped part is certain to contain at least one of these symbols (and at most $p$). Now consider $a^p b^{p+p!}$ for large enough $p$, and mark the $a$ part. We arranged things so that whatever the length of the pumped part (which must consist solely of $a$s), you can pump it to exactly $a^{p+p!} b^{p+p!}$.

Another possibility is to use the Myhill--Nerode theorem. The words $a^n$ ($n \in \mathbb{N}$) are an infinite set of pairwise inequivalent words: if $n \neq m$ then $a^n b^n \notin L$ yet $a^m b^n \in L$.

Take your pick...