If $D$ isn't a convex billiard table, then the billiard map $T$ is not continuous

Question

I am currently reading chapter 3 of "Geometry and Billiards" by Serge Tabachnikov, and I have some doubts about the need of using convex sets.

So, here's how the billiard map is defined:

To fix ideas, consider a plane billiard table $D$ whose boundary is a smooth closed curve $\gamma$. Let $M$ be the space of unit tangent vectors $(x, v)$ whose foot points $x$ are on $\gamma$ and which have inward directions. A vector $(x, v)$ is an initial position of the billiard ball. The ball moves freely and hits $\gamma$ at point $x_1$; let $v_1$ be the velocity vector reflected off the boundary. The billiard ball map $T : M \to M$ takes $(x, v)$ to $(x_1, v_1)$.

So far so good, the definition makes perfect sense to me. Then it is claimed

Note that if $D$ is not convex, then $T$ is not continuous: this is due to the existence of billiard trajectories touching the boundary from inside.

I am confused with this statement. For starters, are we talking with respect to which topology? Should I assume the subspace topology in $\mathbb{R}^2$? And why does not being convex affect continuity? Since the particle in a billiard trajectory starts in the table inside the curve, doesn't it always touch the boundary from the inside? Or am I misinterpreting the justification?

Answer 1

To begin with, note that the "unit tangent vectors $(x,v)$" are not tangent to the curve $\gamma$, but rathert tangent to the plane that $\gamma$ and $D$ live in. That is, $x$ is a point in the boundary set $\partial D$ whihc itself is a subset of $\mathbb R^2$ (and looks nice enough to be parametrized by a sufficiently nice function $\gamma\colon [0,1]\to\mathbb R^2$). The vector $v$ also lives in $\mathbb R^2$ (though by it having unit length, it is restricted to the unit circle $S^1$). Thus all $(x,v)$ are naturally elements of $\mathbb R^4$; hence it is understood that we consider $M$ as subspace of $\mathbb R^2$ and accordingly use the subspace topology. Note that $M$ itself is twodimensional (because $x$ is on the onedimensional $\partial D$ boundary and $v$ on the onedimensional $S^1$), but cannot be viewed as a subspace of §\mathbb R^2$ globally. (It is however homeomorphic to a subspace of the torus).

Once it is clear what topology we consider on $M$, the rest of the questions perhaps becomes clearer. Loosely speaking, if $D$ is not convex, then we can pick a suitable point $x_0\in\partial D$ and let $v$ "swipe" continuously over its allowed values, and it may happen that the value of $T(x_0,v)$ "jumps". Namely, we find points $x_0$ and $x_1$ on $\partial M$ such that the line segment $x_0x_1$ is not completely inside $D$. But we can Also find $x_2$ - e.g., very close to $x_0$ - such that $x_0x_2$ is completely in $D$ (or can we?). Then are are unit vectors $v_1=\frac{x_1-x_0}{|x_1-x_0|}$ and $v_2=\frac{x_2-x_0}{|x_2-x_0|}$ such that $v_i$ points to $x_i$ from $x_0$ (but do the $(x_0,v_i)$ really belong to $M$? I.e., do the $v_i$ necessarily point to the interior from $x_0$?). When $v$ changes continuously from $v_1$ to $v_2$ (why can it? I.e., might there be gaps where $v$ is not pointing to the interior?), then $T(x_0,v)$ moves from $T(x,v_1)$, which is some point $(y_1,w_1)\in M$ such that by construction certainly $y_1\ne x_1$, to $T(x,v_2)$, which is of the form $(x_2,w_2)$. That is, the first component of $T(x_0,v)$ changes from $y_1$ to $x_2$ without ever passing through $x_1$. This shows (how?) that $T$ is not continuous if $D$ is not convex. Or actually, this is just the idea and a very rough sketch of a proof. For a formal proof it leaves a lot of questions open (and I highlighted only very few of them for you to think about).