6

I have met some constrained least square problems, for example, my last post. I found that there are various methods for slightly different constraints, and still I often had little clue about how to solve such problems that I have met.

I have learned some nonlinear optimization and had some good references. I wish to learn more about constrained least square problems, such as

  • what cases admit analytical solutions, and
  • when a case has no analyitical solution, whether the case belongs to convex optimization, and if yes or no, what numerical methods can apply to it the best?

So could someone recommend some references on this topic? Thank you very much!

Royi
  • 10,050
Tim
  • 49,162
  • To the second question: That depends on the constraints: The equality constraints have to be linear to get an convex optimization problem. The inequality constraints should be convex. Then you have a convex optimization problem. Penalty and barrier methods could be interesting for this case...and are applicable in lots of different cases. – Alex Mar 16 '13 at 14:30
  • @steveO Beeing non-convex or non-linear does not mean that you can't solve it with conventional methods such as variational calculus. The main issue is that: If the problem is convex it assures that the max/min found is to be the global one. – AnilB Mar 16 '13 at 16:33
  • 1
    @AnilBaseski: Thanks, (1) but why does variational calculus help to solve constrained least square problems? The variable here is just a vector in $\mathbb R^n$, not a function (which is what calculus of variation is for, isn't it?)? (2) Wait what does "variational calculus" mean? Is it same as calculus of variation? – Tim Mar 16 '13 at 17:41
  • @steveO (2) You are right, variational calculus = calculus of variations (1) The variable can be a vector or tuple or whatever you want. The main thing is that you have the function $f:\Bbb R^n\rightarrow \Bbb R$ and you want to see how the function behaves as the variables change. – AnilB Mar 16 '13 at 17:54
  • @steveO I see what you mean!! You are confused because calculus of variations is used to minimize integrals in general. Sorry my mistake but that is one application area of calculus of variations. – AnilB Mar 16 '13 at 18:01
  • @steveO and Lagrange multipliers and most of the optimization algorithms are just generalizations and extensions of one specific method. http://math.stackexchange.com/questions/331081/simple-resource-for-lagrangian-constrained-optimization/331623#331623 – AnilB Mar 16 '13 at 18:07
  • For the first question, this answer demonstrates that you can hardly get analytical solutions even when your only constraint is positivity. – MathMax Mar 20 '25 at 14:35

1 Answers1

4

There are different ways to define what a convex optimization problem is but most of the time it is one of two things:

  1. it has a convex objective function (as linear least-squares problems do) and linear constraints only (equality and/or inequality). This is a definition that is numerically oriented.

  2. it has a convex objective function, linear equality constraints (if any) and inequality constraints that may be written in the form $c_i(x) \leq 0$ where each constraint function $c_i$ is a convex function. This definition is more general and ensures that the feasible set is a convex set. In turn, this ensures that any local minimizer is also a global minimizer.

So it's easy to determine whether a problem is convex or not.

See my answer to your previous post for references to numerical methods for constrained linear least-squares problems. You'll very rarely find analytical solutions to a nonconvex problem. For convex problems, you know that the KKT conditions characterize any global minimizer. For instance, the KKT conditions of $$ \min \tfrac{1}{2} \|Ax-b\|^2_2 \quad \text{subject to} \ \tfrac{1}{2} \|x\|^2 \leq \tfrac{1}{2} \Delta^2, \ x \geq 0 $$ (where I introduced those $\tfrac{1}{2}$ to cancel out when I differentiate) are: $$ A^T (Ax-b) - \lambda x - z = 0, \ \|x\| \leq \Delta, \ \lambda (\Delta - \|x\|) = 0, \ (x,\lambda,z) \geq 0, \ x_i z_i = 0 \ (i = 1, \ldots, n), $$ where $\lambda$ and $z$ are Lagrange multipliers.

Dominique
  • 3,224