Rather than trying to do something clever, why not just apply the definitions directly? First, we can reasonably assume:
Definition: The absolute value function is the function defined by the formula
$$|x| = \begin{cases} x & \text{if $x \ge 0$, and} \\ -x & \text{ if $x < 0$.} \end{cases} $$
(There are other definitions of the absolute value function, but they are all equivalent to this one over $\mathbb{R}$, and this one is likely the easiest to use in the context of the current question.)
The function of interest then becomes
$$
f(x)
= x^2|x|
= \begin{cases} x^2(x) & \text{if $x \ge 0$, and} \\ x^2(-x) & \text{if $x < 0$} \end{cases}
= \begin{cases} x^3 & \text{if $x \ge 0$, and} \\ -x^3 & \text{if $x < 0$.} \end{cases}
$$
This can be differentiated piecewise using the power rule to get
$$
f'(x)
= \begin{cases} 3x^2 & \text{if $x > 0$, and} \\ -3x^2 & \text{if $x < 0$.} \end{cases}
$$
Note that we can't simply "take the derivative" at zero, since zero is not in the interior of either of the two intervals on which we are differentiating. However, at zero we can directly apply the definition of the derivative:
$$ f'(0)
= \lim_{h\to 0} \frac{f(x+h) - f(x)}{h},
$$
if this limit exists. Taking the limit from the right, we get
$$
\lim_{h\to 0^+} \frac{(0+h)^2|0+h| - 0^2|0|}{h}
= \lim_{h\to 0^+} \frac{h^2|h|}{h}
= \lim_{h\to 0^+} h|h|
= 0.
$$
The limit from the left is essentially the same, from which we may conclude that $f'(0) = 0$. Combining these results, we get
$$
f'(x)
= \begin{cases} 3x^2 & \text{if $x > 0$} \\ 0 & \text{if $x=0$, and} \\ -3x^2 & \text{if $x < 0$.} \end{cases}
$$
By again applying the definition of the absolute value function, this can be simplified to $3x|x|$, which is the claimed result.
