Close to the origin the non-linear terms become small perturbations to the linear "circle" system. The contributions of the perturbation terms accumulate mainly additively, for a finite time and close enough to the origin. Thus if the averaging of the system over one period gives a definitive answer, the same will be true for the original system.
Now in polar coordinates we have
$$
r\dot r=r^4(\cos^4\theta-\cos^2\theta\sin^2\theta)=\frac{r^4}{2}(1+\cos2\theta)\cos(2\theta)=\frac{r^4}{4}(1+2\cos(2\theta)+\cos(4\theta))
\\
r^2\dot\theta=r^2-2r^4\cos^3\theta\sin\theta=r^2-\frac{r^4}2(1+\cos2\theta)\sin2\theta=r^2-\frac{r^4}{4}(2\sin2\theta+\sin4\theta)
$$
Thus on average $\dot{\bar r}=\dfrac{\bar r^3}4$ and $\dot{\bar\theta}=1$. This gives a dynamic that moves away from the origin in forward time, potentially diverging in finite time. But at some point of this divergence the assumption that the third degree terms are small gets violated and other aspects of the global dynamic come to the foreground.
The above means that $\bar r^{-2}$ changes by about $-\pi$ during a full turn, with corrections of order $\bar r^2$, which are negligible as long as $r$ or $\bar r$ are very small. As $\dot \theta=1+O(r^2)$, this also holds up for the leading terms of an approximation of the Poincaré map, thus provides actual proof for the instability of the origin.