Solution using complex numbers. Copied from a deleted question per request.
( update - I have added another solution using circle inversion at end)
Solution 1 - using complex numbers.
Choose a coordinate system so that triangle $ABC$ is lying on the unit circle centered at origin and $A$ on the $x$-axis. Let $a = AM$, $b = BM$, $c = CM$ and $S'$ be the area of a triangle with sides $a,b,c$. In this coordinate system, $S = \frac{3\sqrt{3}}{4}$, we want to show $S' \le \frac{\sqrt{3}}{4}$. Using Heron's formula, this is equivalent to
$$16S'^2 = (a^2+b^2+c^2)^2 - 2(a^4+b^4+c^4) \stackrel{?}{\le} 3$$
Identify euclidean plane with the complex plane. The vertices $A,B,C$ corresponds to $1, \omega, \omega^2 \in \mathbb{C}$ where $\omega = e^{\frac{2\pi}{3}i}$ is the cubic root of unity. Let $z$ be the complex number corresponds to $M$ and $\rho = |z|$, we have
$$
\begin{cases}
a^2 = |z-1|^2 = \rho^2 + 1 - (z + \bar{z})\\
b^2 = |z-\omega|^2 = \rho^2 + 1 - (z\omega + \bar{z}\omega^2)\\
c^2 = |z-\omega^2|^2 = \rho^2 + 1 - (z\omega^2 + \bar{z}\omega)
\end{cases}
\implies
a^2 + b^2 + c^2 = 3(\rho^2 + 1)
$$
Thanks to the identity $\omega^2 + \omega + 1 = 0$, all cross terms involving $\omega$ explicitly get canceled out.
Doing the same thing to $a^4 + b^4 + c^4$, we get
$$\begin{align}a^4 + b^4 + c^4
&= \sum_{k=0}^2 (\rho^2 + 1 + (z\omega^k + \bar{z}\omega^{-k}))^2\\
&= \sum_{k=0}^2\left[ (\rho^2 + 1)^2 + (z\omega^k + \bar{z}\omega^{-k})^2\right]\\
&= 3(\rho^2 + 1)^2 + 6\rho^2\end{align}$$
Combine these, we obtain
$$16S'^2 = 3(\rho^2+1)^2 - 12\rho^2 = 3(1 - \rho^2)^2$$
Since $M$ is inside triangle $ABC$, we have $\rho^2 \le 1$. As a result,
$$S' = \frac{\sqrt{3}}{4}(1-\rho^2) \le \frac{\sqrt{3}}{4} = \frac13 S$$
Solution 2 - using circle inversion.
Let $a = AM, b = BM, c = CM$ again. Let $\Delta(u,v,w)$ be the area of a triangle with sides $u,v,w$. In particular, $S = \Delta(1,1,1)$ and $S' = \Delta(a,b,c)$. We will use the fact $\Delta(u,v,w)$ is homogeneous in $u,v,w$ with degree $2$.
Consider the circle inversion with respect to a unit circle centered at $A$.
Under such an inversion, $B,C$ get mapped to itself while $M$ mapped to a point $M'$ with $$AM' = \frac{1}{a}, BM' = \frac{b}{a}, CM' = \frac{c}{a}$$
We can decompose the quadrilateral $ABM'C$ in two manners. $\triangle ABC + \triangle BM'C$ and $\triangle ABM' + \triangle AM'C$. This leads to
$$\begin{align}
&\verb/Area/(ABC) + \verb/Area/(BM'C) = \verb/Area/(ABM') + \verb/Area/(AM'C)\\
\iff &
S + \Delta(1,\frac{b}{a},\frac{c}{a}) = \Delta(1,\frac{b}{a},\frac{1}{a}) + \Delta(1,\frac{c}{a},\frac{1}{a})\\
\iff &
Sa^2 + S' = \Delta(1,a,b) + \Delta(1,a,c)
\end{align}
$$
By a similar argument, we have
$$
Sb^2 + S' = \Delta(1,b,a) + \Delta(1,b,c)\quad\text{ and }\quad
Sc^2 + S' = \Delta(1,c,a) + \Delta(1,c,b)
$$
Summing these three equalities together and notice
$$\Delta(1,a,b) + \Delta(1,b,c) + \Delta(1,c,a)
= \verb/Area/(ABM) + \verb/Area/(BCM) + \verb/Area/(CAM)
= S$$
We obtain
$$3S' = 2S - S(a^2+b^2+c^2)$$
For any $\triangle XYZ$ and point $P$ in the plane, we know the expression $XP^2 + YP^2 + ZP^2$ is minimized when $P$ is the centroid of $\triangle XYZ$. For an equilateral triangle of side $1$, the centroid is at a distance $\frac{1}{\sqrt{3}}$ from the vertices. This implies $a^2 + b^2 + c^2 \ge 1$.
As a result, $3S' \le S$ and we are done.
