Playing around.
If
$x'(t) = x(x(t))$,
$x''(t)
= x'(t)x'(x(t))
= x(x(t))x(x(x(t)))
$.
By induction,
$x^{(k)}(t)
$
involves terms
with $x(x(...x(t)...)$
nested up to
$k+1$ deep.
If
$x(t)
= \sum_{n=0}^{\infty} a_n t^n
$,
$x'(t)
= \sum_{n=1}^{\infty} na_n t^{n-1}
= \sum_{n=0}^{\infty} (n+1)a_{n+1} t^{n}
$.
We need $x(0) = 0$
for the composition
to be defined.
Therefore
$x'(0)
=x(0)
= 0
$,
$x''(0)
= 0
$.
By induction on
$k$,
$x^{(k)}(0)
=0
$
for all $k$.
Therefore
$x(t)$
can not be a infinite power series.
If
$x(t)
=b t^a
$,
$x'(t)
=abt^{a-1}
$
and
$x(x(t))
=b(bt^a)^a
=b^{a+1}t^{a^2}
$,
so we need
$ab=b^{a+1}$
(so
$a = b^a$ or
$b = a^{1/a})$
and
$a^2=a-1$,
so that
$a
=\frac12(1\pm\sqrt{1-4})
=\frac12(1\pm i\sqrt{3})
$.
For this,
$\begin{array}\\
t^a
&=t^{\frac12(1\pm i\sqrt{3})}\\
&=t^{1/2}e^{\pm \ln(t)\frac12( i\sqrt{3})}\\
&=t^{1/2}\left(\cos(\ln(t)\frac12( \sqrt{3}))\pm i\sin(\ln(t)\frac12( \sqrt{3}))\right)\\
\end{array}
$
From $b = a^{1/a}$
we get $b$,
but I'm not going to work that out.
That is all I can come up with
for now.