Computing the integral
We can start by rewriting the integral in terms of its inverse, similarly as seen in this answer.
$$I=\int_0^{\frac{\pi}{3}}\arccos^2(2\sin^2 x-\cos x) dx \overset{x \to \pi-x}= \int_{\frac{2\pi}{3}}^\pi\arccos^2(2\sin^2 x+\cos x) dx$$
$$\small =\pi^3-\int_0^{\pi^2} \arccos\left(\frac14-\frac14 \sqrt{17-8\cos \sqrt{y}}\right)dy\overset{y\to y^2}=\pi^3-2\int_0^{\pi} y\arccos\left(\frac14-\frac14 \sqrt{17-8\cos y}\right)dy$$
Next, we should notice that the $\arccos$ argument is the "$-$" solution of a quadratic equation with $b=-\frac12$ and $c=\frac{\cos y}{2}-1$ in $y_{\pm}=\frac{-b\pm\sqrt{b^2-4c}}{2}$. Using Vieta's relations we can easily simplify $\arccos y_-\pm\arccos y_+=\arccos\left(y_-y_+\mp\sqrt{(1+y_-y_+)^2-(y_-+y_+)^2}\, \right)$, and therefore, it makes sense to also consider the "$+$" solution like this:
$$\mathcal J_{\pm}=\int_0^\pi y\arccos\left(\frac14\pm\frac14 \sqrt{17-8\cos y}\right)dy$$
Here it should be noted that the "$+$" solution is a purely imaginary number, as the $\arccos$ argument is outside the $(-1,1)$ range, thus we can subtract it from the original integral and directly take the real part.
$$\Rightarrow I= \pi^3-2 \mathcal J_{-}=\pi^3-2\Re\left(\mathcal J_{-}-\mathcal J_{+}\right)= \pi^3-2\Re \int_0^\pi y \arccos\left(-1+\frac12e^{iy}\right)dy$$
$$=4\Re \int_0^\pi y\arcsin\left(\frac12 e^{iy/2}\right)dy=2\Re\sum_{n=0}^\infty \frac{(2n)!}{2^{4n} (n!)^2(2n+1)} \int_0^\pi y e^{iy(n+1/2)}dy$$
$$=4\pi \sum_{n=0}^\infty \frac{(-1)^n(2n)!}{2^{4n} (n!)^2 (2n+1)^2}-8 \sum_{n=0}^\infty \frac{(2n)!}{2^{4n} (n!)^2 (2n+1)^3}$$
$$=8\pi\int_0^1\frac{\operatorname{arcsinh}\left(\frac{y}{2}\right)}{y}dy+16 \int_0^1 \frac{\arcsin\left(\frac{y}{2}\right)\ln y}{y}dy$$
$$=\frac{2\pi^3}{5} - \frac{7\pi^3}{27}=\boxed{\frac{19\pi^3}{135}}$$
Above we used the power series of the $\arcsin$ function (twice) as well as the one for $\operatorname{arcsinh}$. Also, the first integral can be found here and the second one here (after integrating it by parts).
Generalization
It might be worth to mention that we can further attempt to generalize the integral as:
$$I(k)=\int_0^\frac{\pi}{3} \arccos^k\left(2\sin^2 x- \cos x \right) dx=2k\Re \int_0^\pi y^{k-1}\arcsin\left(\frac{1}2 e^{iy/2}\right)dy$$
$$=2k\sum_{n=0}^\infty \frac{(2n)!}{2^{4n} (n!)^2(2n+1)} \Re\left(\frac{1}{i^{k-1}}\frac{d^{k-1}}{dn^{k-1}}\frac{i+e^{in\pi}}{2n+1}\right)$$
For example, with $k=3$ we'll obtain:
$$\small I(3)=\int_0^\frac{\pi}{3} \arccos^3\left(2\sin^2 x- \cos x \right) dx=6\pi^2\sum_{n=0}^\infty \frac{(-1)^n(2n)!}{2^{4n} (n!)^2(2n+1)^2}-48\sum_{n=0}^\infty \frac{(-1)^n(2n)!}{2^{4n} (n!)^2(2n+1)^4}$$
$$\small =12\pi^2\int_0^1\frac{\operatorname{arcsinh}\left(\frac{y}{2}\right)}{y}dy-48 \int_0^1 \frac{\operatorname{arcsinh}\left(\frac{y}{2}\right)\ln^2 y}{y}dy=\frac{3\pi^4}{5}-48_5F_4\left(\begin{array}{c}\tfrac12,\tfrac12,\tfrac12,\tfrac12,\tfrac12\\ \tfrac32,\tfrac32,\tfrac32,\tfrac32\end{array}\middle|-\tfrac14\right)$$
Above there's a huge chance that the last hypergeometric function can be simplified, as we have the closed form for a similar one here:
$$\small _5F_4\left(\begin{array}{c}\tfrac12,\tfrac12,\tfrac12,\tfrac12,\tfrac12\\ \tfrac32,\tfrac32,\tfrac32,\tfrac32\end{array}\middle|\color{red}{\tfrac14}\right) = \frac{\pi}{12}\zeta(3)+\frac{27}{32}\operatorname{Cl}_4\big(\tfrac23\pi\big)$$
Update: Thanks to KStar's efforts, we can indeed obtain a closed form for $I(3)$.
$$I(3)=\boxed{\frac{4\pi^4}{15}-24\zeta(3)\ln(\varphi)+2\pi^2\ln(\varphi)^2-5\ln(\varphi)^4-15\operatorname{Li}_4\left(\frac{1}{\varphi^2}\right)}$$
However, starting from $I(4)$ it seems that the integrals are no longer reducible in closed form, without introducing new special functions.
$$\small I(4)=8\pi^3\sum_{n=0}^\infty \frac{(-1)^n (2n)!}{2^{4n} (n!)^2(2n+1)^2} - 192\pi\sum_{n=0}^\infty \frac{(-1)^n (2n)!}{2^{4n} (n!)^2(2n+1)^4} + 384\sum_{n=0}^\infty \frac{(2n)!}{2^{4n} (n!)^2(2n+1)^5}$$
$$\small = 16\pi^3 \int_0^1 \frac{\operatorname{arcsinh}\left(\frac{y}{2}\right)}{y} dy - 192\pi \int_0^1 \frac{\operatorname{arcsinh}\left(\frac{y}{2}\right) \ln^2 y}{y}dy - 128 \int_0^1 \frac{\arcsin\left(\frac{y}{2}\right) \ln^3 y}{y}dy $$
$$\small =\boxed{\frac{179}{243}\pi^5-96\pi \ln(\varphi)\zeta(3)+8\pi^3\ln^2(\varphi)-20\pi \ln^4(\varphi)-60\pi\operatorname{Li}_4\left(\frac{1}{\varphi^2}\right)-96\operatorname{Gl}_{4,1}\left(\frac{\pi}{3}\right)}$$
Above we introduced the multiple Glaisher function, as seen in $2$ and $5.1$ from this article. In our particular case there is $\operatorname{Gl}_{4,1}\left(\frac{\pi}{3}\right)=\sum_{n=1}^\infty \frac{H_{n-1}}{n^4}\sin\frac{n\pi}{3}$.
