Update a regression on the fly?

Question

Say I have 100 people each with a height, weight, and age. I make a regression that predicts age based on height and weight. Now, I would like to update that model when I meet someone new. I don't want to just re-process 101 people though--I want to take the model that I already have and incorporate the new person into it.

For example, say that I found from the first 100 that:

age = .08*height + .06*weight + 7.

Now, I meet someone with age 120 height 56 and weight 34.

I do know the number of cases with which I originally made the regression. So, my initial idea was that I could just assume that they all fit the model age = .08*height + .06*weight + 7 and somehow weight the new case so that it takes the old into account (e.g., assume I have 100 people that fit the model, so create a regression equation based on the 100 identical data points and then just add the 101? That's almost just like running the regression again on the 101 data points, except that nothing needs to be stored then, because you could derive 100 from the equation.

I would like to do something like this on a large scale, and I don't want to be creating an enormous database of cases, I just want to update the model with each new case.

Answer 1

Let us suppose that you fit a model $Z = a +b X+cY$ based on $N$ data points $(X_i,Y_i,Z_i)$. The so-called normal equations are $$\sum _{i=1}^N Z_i= N a + b\sum _{i=1}^N X_i+ c\sum _{i=1}^N Y_i$$ $$\sum _{i=1}^N X_iZ_i= a\sum _{i=1}^N X_i + b\sum _{i=1}^N X_i^2+ c\sum _{i=1}^N X_iY_i$$ $$\sum _{i=1}^N Y_iZ_i= a\sum _{i=1}^N Y_i + b\sum _{i=1}^N X_iY_i+ c\sum _{i=1}^N Y_i^2$$ and you solve them for $a,b,c$.

If we rewrite these equations in a more symbolic manner, as $$Sz=N a+b Sx + c Sy$$ $$Sxz=a Sx+b Sxx+c Sxy$$ $$Syz=a Sy+b Sxy+c Syy$$ the solutions are given by $$c=\frac{-N \text{Sxx} \text{Syz}+N \text{Sxy} \text{Sxz}+\text{Sx}^2 \text{Syz}-\text{Sx} \text{Sxy} \text{Sz}-\text{Sx} \text{Sxz} \text{Sy}+\text{Sxx} \text{Sy} \text{Sz}}{-N \text{Sxx} \text{Syy}+N \text{Sxy}^2+\text{Sx}^2 \text{Syy}-2 \text{Sx} \text{Sxy} \text{Sy}+\text{Sxx} \text{Sy}^2}$$ $$b=\frac{c N \text{Sxy}-c \text{Sx} \text{Sy}-N \text{Sxz}+\text{Sx} \text{Sz}}{\text{Sx}^2-N \text{Sxx}}$$ $$a=\frac{-b \text{Sx}-c \text{Sy}+\text{Sz}}{N}$$

Now, you add another data point ($N+1$), so the equations are now $$\sum _{i=1}^{N+1} Z_i= (N+1) a + b\sum _{i=1}^{N+1} X_i+ c\sum _{i=1}^{N+1} Y_i$$ $$\sum _{i=1}^{N+1} X_iZ_i= a\sum _{i=1}^{N+1} X_i + b\sum _{i=1}^{N+1} X_i^2+ c\sum _{i=1}^{N+1} X_iY_i$$ $$\sum _{i=1}^{N+1} Y_iZ_i= a\sum _{i=1}^{N+1} Y_i + b\sum _{i=1}^{N+1} X_iY_i+ c\sum _{i=1}^{N+1} Y_i^2$$ But the new sums can be expressed as the old sums plus an extra term corresponding to the new data point $$\sum _{i=1}^{N+1} Z_i=\sum _{i=1}^{N} Z_i+Z_{N+1}$$ $$\sum _{i=1}^{N+1} X_iZ_i=\sum _{i=1}^{N} X_iZ_i+X_{N+1}Z_{N+1}$$ and so on for all the summations.

So, you only need to keep the values of the sums and update them every time you add a new data point. The only thing left is to solve the new three normal equations for which I gave you the formulas.

So, updating the regression by adding one extra point is really simple.

Answer 2

Here I am thinking I should remember something about this, or maybe I shouldn't. We have a matrix "equation": $$ \begin{bmatrix} 1 & h_1 & w_1 \\ \vdots & \vdots & \vdots \\ 1 & h_n & w_n \end{bmatrix} \begin{bmatrix} 7 \\ 0.08 \\ 0.06 \end{bmatrix} \overset{\text{?}} = \begin{bmatrix} a_1 \\ \vdots \\ a_n \end{bmatrix} \tag 1 $$ The equality is not actually equality: on the right, if we put the fitted ages rather than the observed ages, we would have equality. Write $(1)$ in matrix form as $$ X\beta = A. $$ The matrix $X$ has no inverse: there is no $3\times n$ matrix we could put to its right and get the $n\times n$ identity matrix. But there is a left inverse: there is a $3\times3$ matrix we can put to its left to get the $3\times3$ identity matrix. That matrix is $$ \underbrace{(X^T X)^{-1}}_{3\times3} \underbrace{{}\quad X^T\quad{}}_{3\times n}. \tag 2 $$ Hence $$ \begin{bmatrix} 7 \\ 0.08 \\ 0.06 \end{bmatrix} = (X^T X)^{-1} X^T A \tag 3 $$ Exercise: Equality holds in $(3)$ regardless of whether $A$ is the column of fitted ages or the column of observed ages.

Now we add an $(n+1)$th row to $X$, getting $\begin{bmatrix} X \\ W \end{bmatrix}$ where $W\in\mathbb R^{1\times3}$. Now instead of $(2)$ we will have $$ \left(\begin{bmatrix} X^T & W^T \end{bmatrix} \begin{bmatrix} X \\ W \end{bmatrix}\right)^{-1} \begin{bmatrix} X^T & W^T \end{bmatrix} \begin{bmatrix} 7 \\ 0.08 \\ 0.06 \end{bmatrix} \overset{\text{?}} = \begin{bmatrix} a_1 \\ \vdots \\ a_n \\ a_{n+1} \end{bmatrix}. $$

The matrix we need to invert becomes the $3\times3$ matrix $$ \begin{bmatrix} X^T X + W^T W \end{bmatrix}. $$ How to do that without doing the whole thing over from scratch is what I don't know at this moment.

Software experts must have considered this problem, but I don't know what has been done.