Find the stability of $O(0,0)$ for the different values of the parameters a,b.
The system of differential equations is:
$$x'=-x+y+x^2+ax^3$$ $$y'=x-y+ax^2+bxy$$
ATTEMPT:
$F=(F_1,F_2):R^2\rightarrow R^2$
$F(0)=0$, $DF(0)=0$ Consider that:$$x'=-x+y+F_1(x,y)$$ $$y'=x-y+F_2(x,y)$$ with $$F_1(x,y)=x^2+ax^3$$ and $$F_2(x,y)=ax^2+bxy$$
$$\iff$$ $$\begin{bmatrix}x' \\ y' \end{bmatrix}=\begin{bmatrix} -1 &1 \\ 1&-1 \end{bmatrix}+F(x,y)$$
Then we have that :$y(0)=y'(0)=0$ and $x-y+F_2(x,y)$ From these 2 equations we have that $y(x)=\dfrac{-x-ax^2}{bx-1}$ with $b\ne 0$ and $x \ne \dfrac{1}{b}$
For the stationary point i have that $y(0)=0$
So,$$F_1(x,y(x)=\dfrac{-x-ax^2}{bx-1})=-x+\dfrac{-x-ax^2}{bx-1}+x^2+ax^3$$ From this point i don't know how to proceed so to find the kind of stability for the point $O(0,0)$ for the different values of a and b. Am i right until now or no?
I would really appreciate a thorough solution and explanation, since I've just started working on these problems and I have to clear my mind on them.
Thanks in advance !