I'm quite sure that this is a problem from Hirsch, Smale, and Devaney's book in Dynamical systems. So let's take it piece by piece.
a) When $\epsilon = 0$ we have that
$$x ' = 2y(z+1)$$
$$y' = -x(z+1)$$
$$z' = -z^3$$
Now we want to show that the origin is not asymptotically stable. This means there exists some solution that does not tend towards the origin as $t \to \infty$ but stays close to the origin (being a little imprecise here, but you can make it precise by looking at the definition of stable).
Let's try to construct such a solution. For simplicity, let's make the bold assumption that $z = 0$, in which case we now have the equations
$$x' = 2y$$
$$y' = -x$$
$$z' = 0$$
This is basically a $2 \times 2$ system with matrix
$$\begin{bmatrix}
0 & 2 \\
-1 & 0 \\
\end{bmatrix}$$
which has eigenvalues $\pm i \sqrt{2}$. Thus we have solutions to this system of form
$$\begin{bmatrix} x \\ y \end{bmatrix} = C_1 \begin{bmatrix} \cos{t\sqrt{2}} \\ -\sin{t\sqrt{2}} \end{bmatrix} + C_2 \begin{bmatrix} \sin{t\sqrt{2}} \\ \cos{t\sqrt{2}} \end{bmatrix} $$
Putting it together,
$$\begin{bmatrix} x \\ y \\ z \end{bmatrix} = C_1 \begin{bmatrix} \cos{t\sqrt{2}} \\ -\sin{t\sqrt{2}} \\ 0\end{bmatrix} + C_2 \begin{bmatrix} \sin{t\sqrt{2}} \\ \cos{t\sqrt{2}} \\ 0\end{bmatrix} $$
which are ellipses around the origin. These are stable but not asymptotically stable.
b) Here, we can use LaSalle's Invariance Principle. On page 195 (section 9.2 of Hirsch, Smale, and Devaney) we saw that
$$L = x^2+2y^2+z^2$$
is a Liapunov function. Further, we saw that it is a strict Liapunov function since
$$\frac{dL}{dt} = \epsilon(x^2+2y^2)(z+1)-z^4 < 0$$
for $\epsilon < 0$ and $z > -1$.
LaSalle's Invariance Principle given in the textbook says:
Let $X^*$ be an equilibrium point for $X'= F(X)$ and let $L: U \to \mathbb{R}$ be a Liapunov function for $X^*$, where $U$ is an open set containing $X^*$. Let $P \subseteq U$ be a neighborhood of $X^*$ that is closed and bounded. Suppose that $P$ is positively invariant, and that there is no entire solution in $P\setminus\{X^*\}$ on which $L$ is constant. Then $X^*$ is asymptotically stable and $P$ is contained in the basin of attraction of $X^*$.
Here's what we will do. Consider the set
$$P_{abc} = \{(x,y,z) \in \mathbb{R}^3 \quad | \quad |x| \le a, |y| \le b, -1+c \le z \le b$$
Notice that
$$\bigcup_\limits{a > 0, b > 0, 0<c<1} P_{abc} = \{(x,y,z) \in \mathbb{R}^3 \quad | z > -1 \}$$
Furthermore, we have that each $P_{abc}$ contains the origin and is closed. Also, since the Liapunuv function is strict, then each $P_{abc}$ is positively invariant else it would imply that the Liapunov function is increasing (WARNING: THIS IS FALSE. As noted by a comment below, it is NOT true that each of these $P_{abc}$ are positively invariant. I am leaving this solution up here so that you can be warned of my error). Furthermore, having a strict Liapunov function also guarantees that $L$ is not constant along a solution. Thus by LaSalle's Invariance Principle, we must have that each $P_{abc}$ is in the basin of the origin and so the union is also in the basin.