Based on @noam.szyfer's answer and our discussion in the comment section under his post, I would like to propose the following solution to improve the rigorousness. I really appreciate @noam.szyfer's help!
We want to prove
\begin{align*}
\mu_*(A)\leq\mu^*(A)\ \text{holds for each subset $A$ of $X$}.
\end{align*}
Assume to the contrary that $\mu_*(A)>\mu^*(A)$ for some $A\in X$. If $\mu^*(A)=+\infty$, then we have already reached a contradiction. So assume that $\mu^*(A)<+\infty$. Note that it follows from this assumption that there exists a $B_1 \in \mathcal{A}$ such that $A \subseteq B_1$ and $\mu(B_1)<+\infty$.
Without loss of generality, suppose that $\mu_*(A)=\mu^*(A)+\epsilon$ for some $A\in X$ and $\epsilon>0$. Since $\mu^*(A)$ is the greatest lower bound of the set $\{\mu(B):A \subseteq B\ \text{and}\ B\in\mathcal{A}\}$, it follows that there is a $B_1\in\mathcal{A}$ such that $A \subseteq B_1$ and
\begin{align*}
\mu(B_1)<\mu^*(A)+\frac{\epsilon}{2}.\tag1
\end{align*}
For otherwise, if $\mu(B_1)\geq\mu^*(A)+\frac{\epsilon}{2}$ for all $B_1 \in \mathcal{A}$ such that $A \subseteq B_1$, then $\mu^*(A)$ would not be the greatest lower bound of the set $\{\mu(B):A \subseteq B\ \text{and}\ B\in\mathcal{A}\}$, because $\mu^*(A)+\frac{\epsilon}{2}$ would be a lower bound bigger than $\mu^*(A)$.
Similarly, $\mu_*(A)$ is the least upper bound of the set $\{\mu(B):B \subseteq A\ \text{and}\ B \in \mathcal{A}\}$. If $\mu_*(A)=+\infty$, then there exists a $B_2\in\mathcal{A}$ such that $B_2 \subseteq A$ and $\mu(B_2)=+\infty$. But $B_2 \subseteq A \subseteq B_1$ and $B_1,B_2\in\mathcal{A}$ would imply that $\mu(B_2)\leq\mu(B_1)$, which is impossible because $\mu(B_1)<+\infty=\mu(B_2)$. So we obtain that $\mu_*(A)<+\infty$. Note that the finiteness of $\mu_*(A)$ implies the finiteness of $\epsilon$. It then follows that there is a $B_2\in\mathcal{A}$ such that $B_2 \subseteq A$ and
\begin{align*}
\mu_*(A) < \mu(B_2) + \frac{\epsilon}{2},\tag2
\end{align*}
which (since $\epsilon<+\infty$) implies
\begin{align*}
\mu_*(A) - \frac{\epsilon}{2} < \mu(B_2).
\end{align*}
For otherwise, if $\mu_*(A)-\frac{\epsilon}{2}\geq\mu(B_2)$ for all $B_2\in\mathcal{A}$ such that $B_2 \subseteq A$, then $\mu_*(A)$ would not be the least upper bound of the set $\{\mu(B):B \subseteq A\ \text{and}\ B \in \mathcal{A}\}$, because $\mu_*(A)-\frac{\epsilon}{2}$ would be an upper bound smaller than $\mu_*(A)$.
Hence, by inequalities (1) and (2), we have
\begin{align*}
\mu(B_1) + \mu_*(A) < \mu^*(A) + \mu(B_2) + \epsilon,
\end{align*}
which (by plugging in our assumption $\mu_*(A)=\mu^*(A)+\epsilon$) implies
\begin{align*}
\mu(B_1)+\mu^*(A)+\epsilon < \mu^*(A)+\mu(B_2)+\epsilon.
\end{align*}
Since $\mu^*(A)$ and $\epsilon$ are both finite, it follows that
\begin{align*}
\mu(B_1)<\mu(B_2).
\end{align*}
However, since $B_2 \subseteq A \subseteq B_1$ and $B_1,B_2\in\mathcal{A}$ imply that $\mu(B_2)\leq\mu(B_1)$, we reached a contradiction. Therefore, we have proved that $\mu_*(A)\leq\mu^*(A)$ holds for each subset $A$ of $X$.