I better clarify my question. If the problem of RSA is finding the inverse of $e^{-1} = e$ in:
$a^e \mod N$
Why don't we just do:
$ae \mod N$
for encryption, and
$ae^{-1} \mod N$
for decryption?
Since I'm quite sure that the answer is: "because the former formula is attackable", it would be really helpful at least a resource to understand such attack.
Edit: I'm going to answer my own question, thanks to everyone has answered, I apologize if I was a little bit rude in the comments, but you probably didn't get my question either (even if some answers were part of mine below).
My assumption was that multiplicative modular inverses $\mod N$ were the same as the ones $\mod \phi(N)$ but it's clearly wrong. My assumption was to multiply by $e^{-1} \mod \phi(N)$, but the problem is that since the inverses of a number $\mod N$ and $\mod \phi(N)$ are different, this multiplication wouldn't 'produce' $a$. i.e.
$ae^{-1}e \mod N \neq a$
if
$e$ and $e^{-1}$ are inverses $\mod \phi(N)$ and not $\mod N$
Also, if we proceed taking $e, e^{-1} \mod N$, as someone has already replied, would restrict the security of RSA to compute the EEA (Extendend Euclid Algorithm) which is way faster to compute compared to the factorization, hence breaking the security of the protocol.