I'm trying to make a gradient descent algorithm where the parameters need to be constrained by a set of linear equations. So if $\theta_i$ are parameters and I'm trying to minimize $E$, then $$ \theta_i \rightarrow \theta_i - \alpha \nabla_{\theta_i} E $$ except that $\theta_i$ must also obey more than one linear equations of the form $$ \sum_i \theta_i c_i = Z $$
I don't want to add a penalty function to $E$ because I want this to be an exact constraint. I'm wondering if anyone can point me to a method that could do this.
I'll briefly describe a vague idea I have, which is to leave $\theta_i$ unconstrained and then transform them to $\tilde{\theta_i}$, which would follow the constraints and be input to $E$. So $$ \theta_i \rightarrow \theta_i - \alpha \nabla_{\theta_i} E(\tilde{\theta_i}) $$ $$ \sum_i \tilde{\theta_i} c_i = Z $$
My idea for the transformation would be to remove the component of $\theta_i$ spanned by the vectors $c_i$ in the constraint equations, then solve the linear constraint equations in this space, and add the solution back to $\theta_i$, making $\theta_i$ free to move in the space not spanned by the constraint vectors. However I can't think of a straight forward way to do this.
