Set $x=\begin{bmatrix} x_1 \\ x_2\end{bmatrix}, A=\begin{bmatrix} 1 & 0 \\ 2 & -3\end{bmatrix}$ and $b(t)=\begin{bmatrix} 10\cos (t) \\ 0\end{bmatrix}$, for all $t\in \mathbb R$.
Your system is equivalent to $x'=Ax+b$.
Given $t\in \mathbb R$, it can be rewritten as $$\begin{cases} x_1'(t)&=x_1(t)+ 10\cos (t)\\ x_2'(t)&=2x_1(t)-3x_2(t)\end{cases}$$
The equation $x_1'(t)=x_1(t)+ 10\cos (t)$ is just an ODE. It can be solved using the standard methods, (for instance this one). The general solution of this equation is determined by $$\forall t\in \mathbb R\left(x_1(t)=5\sin(t)-5\cos(t) +C_1e^t\right),$$ for some $C_1\in \mathbb R$.
So now you can just replace $x_1(t)$ by the above expression in the second equation yielding:
$$x'_2(t)=10\sin(t)-10\cos(t)+2C_1e^t-3x_2(t).$$
This, again, is just an ODE. The solution is given by, for some $C_2\in \mathbb R$, $$\forall t\in \mathbb R\left(x_2(t)=2\sin(t)-4\cos(t)+\dfrac{C_1}{2}e^t+C_2e^{-3t}\right).$$
A particular solution to the system can be found setting $C_1=C_2=0$ and this is exactly the solution you found.
Since you're looking to solve this with the indeterminate coefficients method, let me also do this.
Take the ansatz $x_p=B\cos +C\sin $, where $B,C\in \mathbb R^{2\times 1}$ are constants.
Write $B=\begin{bmatrix} b_1\\ b_2\end{bmatrix}, C=\begin{bmatrix} c_1\\ c_2\end{bmatrix}$ and $x_p(t)=\begin{bmatrix} b_1\cos(t)+c_1\sin (t)\\ b_2\cos(t)+c_2\sin(t)\end{bmatrix}$
For all $t\in \mathbb R$ set $$\begin{bmatrix} -b_1\sin(t)+c_1\cos (t)\\ -b_2\sin(t)+c_2\cos(t)\end{bmatrix}=x_p'(t)=Ax_p(t)+b(t)=\begin{bmatrix} b_1\cos(t) + c_1\sin(t)+10\cos(t)\\ (2b_1-3b_2)\cos(t)+(2c_1-3c_2)\sin(t)\end{bmatrix},$$
which is equivalent to $$\begin{cases} (10+b_1-c_1)\cos(t)+(c_1+b_1)\sin(t)&=0\\ (2b_1-3b_2-c_2)\cos(t)+(2c_1-3c_2+b_2)\sin(t)&=0\end{cases} \tag {$\spadesuit$}$$
Since $\left(\spadesuit\right)$ holds for all $t$, it holds in particular for $t=0$ and $t=\dfrac\pi 2$ thus providing the system $$\begin{cases} b_1+ c_1&=0\\ b_1-c_1&=-10\\ 2b_1-3b_2-c_2&=0\\ b_2+2c_1-3c_2&=0\end{cases}$$
whose solution is attained with $(b_1, b_2, c_1, c_2)=(-5, -4, 5, 2)$.
Therefore $x_p(t)=\begin{bmatrix} -5\cos(t)+5\sin (t)\\ -4\cos(t)+2\sin(t)\end{bmatrix},$ for all $t\in \mathbb R$.
x_p_(t) = {{-5},{-4}}cos(t) + {{5},{2}}sin(t)
Is this correct?
– TheColonel May 01 '14 at 00:33