Combining two fundamental matrices

Question

Let $\mathcal{F_{ab}}$ be the fundamental matrix obtained from images $A$ and $B$

$$ \mathcal{F_{ab}} = \begin{bmatrix} ab_{11} & ab_{12} & ab_{13} \\ ab_{21} & ab_{22} & ab_{23} \\ ab_{31} & ab_{32} & ab_{33} \\ \end{bmatrix} $$

and let $\mathcal{F_{bc}}$ be the fundamental matrix obtained from image $B$ and $C$

$$ \mathcal{F_{bc}} = \begin{bmatrix} bc_{11} & bc_{12} & bc_{13} \\ bc_{21} & bc_{22} & bc_{23} \\ bc_{31} & bc_{32} & bc_{33} \\ \end{bmatrix} $$

and let $\mathcal{F_{ac}}$ be the fundamental matrix obtained from images $A$ and $C$

$$ \mathcal{F_{ac}} = \begin{bmatrix} ac_{11} & ac_{12} & ac_{13} \\ ac_{21} & ac_{22} & ac_{23} \\ ac_{31} & ac_{32} & ac_{33} \\ \end{bmatrix} $$

Is it possible to get $\mathcal{F_{ac}}$ as a function of $\mathcal{F_{ab}}$ and $\mathcal{F_{bc}}$?

NOTES :

• $A,B \text{ and } C$ are images.
• $ab, bc \space and \space ca$ are just component variable of Fundamental Matrices [Video for Explanation] gained from the images. $ab$ is not product of $a$ and $b$, it's just variable name.
• The numeric subscripts $(_{ij})$ are used for indicating positions.
• Image is a matrix whose size is like [height, width] and Fundamental Matrix is gained from two images.
• The Fundamental Matrix can be generated from Image Matrix using OpenCV. The Source Code of Function cvFindFundamentalMat is here.

Answer 1

If you see the fundamental matrix as a kind of transformation that sends points in a plane to what we call epipolar lines in another plane we ask if when we have three images, call them A, B, and C, is the same to send a point directly than to send a point to a halfway point and send this point to the final destination.

Intuitively this is the same as composing two transformations. Therefore we will check:

$$ \mathcal{F}_{AC}=\mathcal{F}_{AB}\mathcal{F}_{BC}$$

Remember that the fundamental matrix $\mathcal{F}_{AC}$ on the left sends points from A to C and on the right sends points from C to A. Now we know that the fundamental matrix is linked to the essential matrix by the equation:

$$\mathcal{F}_{AB} = \mathcal{K}_A^{-T} \mathcal{E}_{AB}\mathcal{K}_B^{-1}$$

Where $\mathcal{E}_{AB}$ represents the essential matrix of the images A and B and $\mathcal{K}_{A}$ and $\mathcal{K}_{B}$ is the calibration matrices associated with the cameras A and B respectively. Using this relation in the previous equation we have:

$$\mathcal{E}_{AC} = \mathcal{E}_{AB}\mathcal{K}^{-T}_B\mathcal{K}_B^{-1}\mathcal{E}_{BC}$$

It's easy to see that, in general, $\mathcal{K}^{-T}_B\mathcal{K}_B^{-1}\neq I$. Let's assume that the camera of image B is calibrated then $\mathcal{K}_B=1$ and check if:

$$\mathcal{E}_{AC} = \mathcal{E}_{AB}\mathcal{E}_{BC}$$

As the camera attached to image B is calibrated suppose that the projection matrices are given in the coordinate system attached to B. If we denote by $\mathcal{R}_k$ and $t_k$ the rotation matrix and the translation vector of the camera associated with the image $k=$A, C with respect to the first camera we have that

$$\mathcal{E}_{BC} = [-t_C]_\times \mathcal{R}_C^T $$ $$\mathcal{E}_{AB} = [t_A]_\times \mathcal{R}_A $$ $$\mathcal{E}_{AC} = [t_A-t_C]_\times \mathcal{R}_A \mathcal{R}_C^T $$

Using this relation in our equation and adjusting the rotation matrices and the translation matrices, we obtain

$$ [t_A-t_C]_\times = [t_A]_\times \mathcal{R}_A [-t_C]_\times \mathcal{R}_A^T$$

The right part of the equation is equal to zero but the left part is not equal to zero, as B$\neq$C. Then we have that we can't get $\mathcal{F}_{AC}$ as a function of $\mathcal{F}_{AB}$ and $\mathcal{F}_{BC}$