This is a partial answer. I only get an integral representation for the expected area.
We will use the fact:
Given $3$ positive numbers $a,b,c$, they are the sides of a non-degenerate triangle if and only if we can find $3$ positive numbers $u,v,w$ such that
$$a = v + w,\;b = u + w,\;c = u + v$$
Notice $da \wedge db \wedge dc = 2 du \wedge dv \wedge dw$, the Jacobian is a constant. Uniform distribution for $a,b,c$ over unit-cube is equivalent to uniform distribution for $u,v,w$ over corresponding polyhedron in $(u,v,w)$-space:
$$\Omega = \left\{ (u,v,w) \in \mathbb{R}^3 : 0 \le v + w, u + w, u + v \le 1 \right\}$$
Let $\Omega_{+} = \Omega \cap [0,\infty)^3$ and $\Delta = \left\{ (u,v,w) \in \Omega_{+}, u \ge v \ge w \right\}$.
In terms of $u,v,w$, the probability of forming a triangle is
$$P_\triangle = 2 \int_{\Omega_{+}} du dv dw = 12\int_{\Delta} du dv dw$$
Change variable to $(u,v,w) = (u,us,ust)$, the condition $u \ge v \ge w$ is equivalent to $s,t \in [0,1]$ and
$$P_\triangle = 12 \int_0^1\int_0^1\left(\int_0^{\frac{1}{1+s}} u^2 du\right) sds dt = 4 \int_0^1 \frac{sds}{(1+s)^3} = \frac12
$$
In terms of $u,v,w$, the area of triangle $A = \sqrt{uvw(u+v+w)}$. Its expected area is given by the formula
$$\begin{align}
\verb/E/[A] &= \frac{12}{P_\triangle}\int_\Delta \sqrt{uvw(u+v+w)} du dv dw\\
&=24 \int_0^1\int_0^1 \left(\int_0^{\frac{1}{1+s}} u^4 du\right)\sqrt{t(1+s+st)} s^2ds dt\\
&= \frac{24}{5}\int_0^1\int_0^1 \frac{s^2\sqrt{t(1+s+st)}}{(1+s)^5} ds dt
\end{align}
$$
I don't know how to evaluate the last integral. However, wolfram alpha and maxima 5.17.1 evaluate the integral in last line numerically as
$0.0241652$ and $0.0241651708563157 \pm 2.68\times 10^{-16}$ respectively.
This leads to $\verb/E/[A] \sim 0.11599282\ldots$ matching what you get from simulation.
Update
According to @Max0815 in comment, Mathematica do know how to evaluate the integral in close form. The result is:
$$\verb/E/[A] = \frac1{1920}\left(
\small
\begin{align}
& 1632 - 720\sqrt3 - 576 C
\\& + 5\sqrt3\left(\psi^{(1)}\!\!\left(\frac16\right) + \psi^{(1)}\!\!\left(\frac13\right) - \psi^{(1)}\!\!\left(\frac23\right) - \psi^{(1)}\!\!\left(\frac56\right)\right)
\end{align}
\right)\\
=\frac{1}{80}\left(68 - 30\sqrt3 - 24 C + 15\operatorname{Cl}_2\left(\frac\pi3\right)\right) \phantom{======...}
$$
where $C$ is the Catalan's constant and $\psi^{(1)}$ is the $1^{st}$ derivative of digamma function.
I have no idea how to get this analytically. However,
WA evaluates this numerically to $$0.1159928201103153651023020463820551827782699626985368352024021543\ldots$$
which does look right.
$$ \int_0^\frac12\int_a^{1-a}\int_{a+b}^1\mathrm dc,\mathrm db,\mathrm da=\frac1{12} $$
and the volume with $c\le2a$ (which implies $c\le a+b$) is
$$ \int_0^\frac12\int_a^{2a}\int_b^{2a}\mathrm dc,\mathrm db,\mathrm da+\int_\frac12^1\int_a^1\int_b^1\mathrm dc,\mathrm db,\mathrm da=\frac1{48}+\frac1{48}=\frac1{24};. $$
– joriki Jan 13 '24 at 16:04