We can identify $x\in \mathfrak{so}(3)$ with $\boldsymbol v \in \mathbb R^3$:
$$ \begin{bmatrix} 0 & -v_3 & v_2 \\ v_3 & 0 & -v_1 \\ -v_2 & v_1 & 0 \end{bmatrix} \mapsto \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix} $$
with the cross product as the Lie bracket.
I remember hearing that every non-zero endomorphism of $\mathbb R^3$ that preserves the cross product is of the form
$$\boldsymbol v \mapsto R \boldsymbol v .$$
where $R \in SO(3)$. This corresponds to
$$x \mapsto R^T x R.$$
But
$$x \mapsto xB + Bx$$
corresponds to
$$\boldsymbol v \mapsto (\text{tr}(B)I - B)\boldsymbol v.$$
Now we use that the endomorphism is symmetric. It seems to me that the Killing form on $\mathbb R^3$ with the cross product must be the standard inner product. So it follows that $R$ is symmetric.
That leaves two possibilities: $R = I$, when $B = \frac12 I$ works, or $R = I - 2 \boldsymbol n \otimes \boldsymbol n$ with $\|\boldsymbol n\| = 1$, in which case $B = -\frac12 I +2 \boldsymbol n \otimes \boldsymbol n$ works.
Added later: Why is a non-zero endomorphism on $\mathbb R^3$ that preserves the cross product necessarily an element of $SO(3)$? So suppose the endormorphism is $\boldsymbol v \mapsto R \boldsymbol v$. Let the three columns of $R$ be $\boldsymbol a$, $\boldsymbol b$, and $\boldsymbol c$. Then we have
$$ \boldsymbol a \times \boldsymbol b = \boldsymbol c, \quad
\boldsymbol b \times \boldsymbol c = \boldsymbol a, \quad
\boldsymbol c \times \boldsymbol a = \boldsymbol b . $$
Now if $\boldsymbol a$ and $\boldsymbol b$ are linearly dependent, then $\boldsymbol c = \boldsymbol 0$, from which it follows that $\boldsymbol a = \boldsymbol b = \boldsymbol 0$, which contradicts that the endomorphism is non-zero.
Now consider:
$$ \boldsymbol a \times (\boldsymbol a \times \boldsymbol b) = \boldsymbol a \times \boldsymbol c = - \boldsymbol b,$$
and
$$ \boldsymbol a \times (\boldsymbol a \times \boldsymbol b) = (\boldsymbol a \cdot \boldsymbol b) \boldsymbol a - \|\boldsymbol a\|^2 \boldsymbol b .$$
Then we see that $\boldsymbol a \cdot \boldsymbol b = 0$ and $\|\boldsymbol a\| = 1$. Similarly for any other pair of them. Thus $\boldsymbol a$, $\boldsymbol b$ and $\boldsymbol c$ are orthogonal unit vectors. Furthermore, they form a right handed pair. So $R \in SO(3)$.
Note: If $B$ were positive definite, then $\text{tr}(B)I - B$ could be a moment of inertia matrix created from the second moment tensor
$$ B = \int_{\mathbb R^3} \rho(\boldsymbol r) \boldsymbol r \otimes \boldsymbol r \, d \boldsymbol r $$
(here $\rho$ is the density function), and thus the map
$$x \mapsto xB + Bx$$
is really a map from angular acceleration to angular momentum.