Since you already know that $E$ contains an interval, it will suffice to prove the following.
Lemma. If a subset $E$ of $\mathbb R$ is "midpoint-convex" and has nonempty interior, then $E$ is an interval.
Proof. Let $I$ be a maximal interval contained in $E$, and assume for a contradiction that $E\ne I$. Then there is a point $a\in E$ such that $a\lt\inf I$ or $a\gt\sup I$; without loss of generality we assume that $a\lt c=\inf I$. Choose $d\in I$ so that $c\lt d$; then $(c,d)\subseteq E$.
By repeated application of midpoint-convexity, we can find points in $E\cap(a,c)$ which are arbitrarily close to $c$. Choose a point $e\in E\cap(a,c)$ which is closer to $c$ than $c$ is to $d$.
Since $e\in E$ and $(c,d)\subseteq E$, it follows by midpoint-convexity that $\left(\frac{e+c}2,\frac{e+d}2\right)\subseteq E$. Since $\frac{a+c}2\lt c\lt\frac{a+d}2$, it follows that the set
$$J=\left(\frac{e+c}2,\frac{e+d}2\right)\cup I$$
is an interval. Since $I\subsetneqq J\subseteq E$, this contradicts the maximality of $I$.