I'm trying to analyze the asymptotic order of $n^{\ln n}$ and $(\ln n)^n$ At first, I take $\ln$ to both-hand-sides. So I got $(\ln n)^2$ and $n\ln(\ln n))$. However, I don't know what I should do next.
Actually, my friends said, substitute $n = e^k$, but I'm not sure about this approach. How can I analyze these two expressions?
Considering $n=\mathrm e^k$ is not necessary.
By the step you explain, one wants to compare $\exp((\ln n)^2)$ and $\exp(n\ln\ln n)$. Every power of $\ln n$ is negligible when compared to every (positive) power of $n$,hence $(\ln n)^2\ll n\ll n\ln\ln n$. This is more than enough to conclude that $\exp((\ln n)^2)\ll\exp(n)\ll\exp(n\ln\ln n)$ since these last comparisons are in fact equivalent to $n-(\ln n)^2\to+\infty$ and $n\ln\ln n-n\to+\infty$.
To sum up, $n\ln\ln n-(\ln n)^2\to+\infty$ hence $n^{\ln n}=\exp((\ln n)^2)\ll\exp(n\ln\ln n)=(\ln n)^n$, in the sense that $$ \lim_{n\to\infty}\frac{n^{\ln n}}{(\ln n)^n}=0. $$
A more general technique applies here. If you have two functions $s(n) < b(n)$ then under some minor additional conditions you'll have $$ s(n)^{b(n)} > b(n)^{s(n)} $$ In loose terms, "small-to-the-big beats big-to-the-small." For a proof of this, refer to this answer.
