I have proved $P'=AP$
where
$$P= \begin{pmatrix} T \\ N \\B \end{pmatrix}$$
$$A= \begin{pmatrix} 0 & \kappa & 0 \\ -\kappa & 0 & \tau \\ 0 & -\tau & 0 \\ \end{pmatrix} $$.
I am trying to show $P$ is orthogonal by this fact. I try to differentiate $PP^t$, then $P'(P^t)+P(P^t)'=APP^t-PP^tA$ since $A$ is skew-symmetric. But I am stuck here, I am wondering if it is true that $APP^t=PP^tA$.
What you cannot do: You can't show just from $P' = AP$ that $P$ is orthogonal. For example when $\kappa, \tau$ are both identically zero, then one has $P' = 0$, so $P$ is constant. Thus if $P$ is not orthogonal at $t=0$, it is not for all $t$.
What you can do: If $P$ is orthogonal at time $t=0$, then the same is true for all $t$: you derived the initial value problem (with $B = PP^t$) $$ B' = AB - BA,\ \ \ B(0) = I.$$ Since $B_1(t) = I$ is clearly a solution to the above IVP, it must be the solution (by the uniqueness theorem in ODE). So $PP^t = I$ for all $t$.