if you look at the non trivial boundary, i mean $u_x(a,y)$, you can see it has functionality of $y$ (as in general it should have) while $x'_n(a)$ is just a number! so this assignment
$$
x'_n(a) = \frac{T \sin^3(\pi y/a)}{\sin(n \pi y/b)}
$$
is either completely wrong or just mean the fraction in right hand side is a constant which doesn't seem true! so this is the start of where you should know something is wrong.
for solving this problem you should finish construction of
$u(x,y)$ completely and then try to use non-trivial boundary condition. so till the point
$$
x_n(x) = c_n \sinh (\frac{n\pi x}{b})
$$
everything is alright. (
$c_n$ is still an unknown constant so
$2c_n$ or
$c_n$ doesn't matter)
so general representation of
$u(x,y)$ would be:
$$
u(x,y) = \sum_{n \in \mathbb{Z}} A_n u_n(x,y) =\sum_{n \in \mathbb{Z}} A_n x_n(x)y_n(y) \\
= \boxed {\sum_{n \in \mathbb{Z}} A_n \sinh (\frac{n\pi x}{b}) \sin(\frac{n\pi y}{b})}. \tag{1}\label{1}
$$
therefore the problem is finding
$A_n$. we also have
$$
u_x(x,y) = \sum_{n \in \mathbb{Z}} A_n (\frac{n\pi}{b}) \cosh(\frac{n\pi x}{b}) \sin(\frac{n\pi y}{b}) = \frac{\pi}{b}\sum_{n \in \mathbb{Z}} n.A_n \cosh(\frac{n\pi x}{b}) \sin(\frac{n\pi y}{b}).
$$
Now we should use condition
$u_x(a,y)$:
$$
u_x(a,y) = \frac{\pi}{b}\sum_{n \in \mathbb{Z}} n.A_n \cosh(\frac{n\pi a}{b}) \sin(\frac{n\pi y}{b}) = T \sin^3(\frac{\pi y}{a}) \tag{2}\label{2}
$$
for solving this you should use orthoganality relation of
$y_n(y)=\sin(\frac{n\pi y}{b})$ which is :
$$
\int_0^b y_n(y) y_m(y) dy = \frac{b}{2} \delta_{n,m}
$$
where
$\delta_{n,m}$ is
kroneker delta . So multiply
$\eqref{2}$ by
$y_m(y)=\sin(\frac{m\pi y}{b})$ and integrate it from
$0$ to
$b$ :
$$
\frac{\pi}{b}\sum_{n \in \mathbb{Z}} n.A_n \cosh(\frac{n\pi a}{b}) (\frac{b}{2} \delta_{n,m}) = T \int_0^b \sin(\frac{m\pi y}{b}) \sin^3(\frac{\pi y}{a}) dy \\
A_m \frac{m \pi}{2} \cosh(\frac{m\pi a}{b}) = T \int_0^b \sin(\frac{m\pi y}{b}) \sin^3(\frac{\pi y}{a}) dy .
$$
Hence
$$
\boxed {A_n = \frac{2T}{n\pi \cosh(\frac{n\pi a}{b}) } \int_0^b \sin(\frac{n\pi y}{b}) \sin^3(\frac{\pi y}{a}) dy}. \tag{3}\label{3}
$$
So
$u(x,y)$ is expanding as
$\eqref{1}$ with
$A_n$ defined as
$\eqref{3}$.
I'm pretty sure the integral
$\eqref{3}$ should have explicit form but in case of
$a$ be the denominator in
$\sin^3$ it should be very complicated.
