You can decompose your rectangle. I note the point with coordinates, $A=(0,0)$ and $B=(4,4)$, first coordinate is horizontal. I note $N(x_1,..,x_n,\neg y_1, .., \neg y_m)$ the number of paths that go through all the $x_i$ but don't go through the $y_i$.
Since any shortest path going through $(2,0)$ or $(2,1)$ will go through $(3,1)$ or $(4,1)$, we have
$N=N((4,1)) + N((3,1),\neg (4,1)) + N(\neg (2,0),\neg (2,1))$
$N((4,1))=\frac{5!}{4!1!}=5$ according to your formula, since once we are in $(4,1)$, there is only one way to go to $B$
Since $(3,2)$ is the only one node after $(3,1)$ that is not $(4,1)$, and there are only two distinct path from $(3,2)$, we have
$N((3,1),\neg (3,1)) = N((3,2),\neg (2,2)) = N((3,2),(3,4))+N((3,2),(4,2))=2 \times \frac{4!}{3!1!}=8$.
Now, note that $N(\neg (2,0), \neg (2,1)) = N((1,1),\neg (2,1)) + N((0,2))$...
I'm a rather lazy guy, so I won't finish this, it is quit exhaustive.
I think, (by calculating the rest mentally) it is $N=25$, but you need to do the complete thing in order to be sure.
Rq : I think you can do a similar reasoning with edges instead of point, it may be faster.