This is true. We need to repeatedly use the following key lemma:
Key lemma: If $T^{\ast}$ is diagonalizable, it remains diagonalizable after quotienting by any invariant subspace of $V^{\ast}$. Taking the contrapositive, if $T^{\ast}$ is not diagonalizable on some such quotient then it is not diagonalizable.
We'll also need to use the fact that the action of $T$ equips both $V$ and $V^{\ast}$ with the structure of a $K[T]$-module, where $K$ is the underlying field, and that in terms of the module structure, a linear operator $X$ acts diagonalizably iff the induced $K[X]$-module is a direct sum of modules of the form $K[X]/(X - \lambda)$ (this is a slightly stronger condition than the condition that the module is semisimple).
The argument splits into a few cases. We need to use the axiom of choice below, although it is heavily disguised, in order to conclude that if $f : U \to W$ is an injection then $f^{\ast} : W^{\ast} \to U^{\ast}$ is a surjection, so that nontrivial $K[T]$-submodules of $V$ do in fact correspond to nontrivial $K[T]$-quotient modules of $V^{\ast}$.
Case 1: $T$ has either 1) a generalized eigenvector which is not an eigenvector, or 2) an eigenvector over a field extension of $F$ whose eigenvalue is not in $F$. Then $V$ contains $F[T]/f(T)$ as a submodule where either $f(T) = (T - \lambda)^2$ or $f(T)$ is irreducible of degree greater than $1$ over $F[T]$. It follows that $V^{\ast}$ also contains $F[T]/f(T)$ as a quotient module, hence $T^{\ast}$ does not act diagonalizably on this quotient.
Case 2: $T$ has infinite spectrum. Then $V$ contains $\bigoplus_{\lambda} F[T]/(T - \lambda)$ as a submodule, where the sum runs over some infinite set of distinct eigenvalues. It follows that $V^{\ast}$ admits $\prod_{\lambda} F[T]/(T - \lambda)$ as a quotient module, so it suffices to show that $T^{\ast}$ acting on such an infinite product is not diagonalizable.
But it's clear that the direct sum of the eigenspaces of $\prod_{\lambda} F[T]/(T - \lambda)$ is $\sum_{\lambda} F[T]/(T - \lambda)$ and that this is a proper submodule, since for example it does not contain $\prod_{\lambda} 1$.
Case 3: $T$ has a torsion-free vector (a vector $v \in V$ generating a submodule isomorphic to $F[T]$; in particular, a vector which cannot be written as a sum of eigenvectors). Then $V$ contains $F[T]$ as a submodule. It follows that $V^{\ast}$ admits $F[T]^{\ast}$ as a quotient module, so it suffices to show that $T^{\ast}$ acting on $F[T]^{\ast}$ is not diagonalizable.
$F[T]^{\ast}$, as an $F[T]$-module, can be explicitly described as exponential (divided power) power series
$$F \langle \langle S \rangle \rangle = \left\{ \sum a_k \frac{S^k}{k!} : a_k \in K \right\}$$
(the factorial divisions are just notation, this works in any characteristic) with $T^{\ast}$ acting by the derivative $\frac{\partial}{\partial S}$. The isomorphism with $F[T]^{\ast}$ sends $\frac{S^k}{k!}$ to the linear functional $e_k : F[T] \to F$ which extracts the $k^{th}$ coefficient of a polynomial; this corresponds to thinking of $\frac{S^k}{k!}$ as acting by $\frac{1}{k!} \frac{\partial^k}{\partial T^k}$ together with evaluation at $0$. It follows from this identification that $T^{\ast}(e_k) = e_{k-1}$, hence the isomorphism with the derivative.
This means the eigenvectors of $T^{\ast}$ acting on $F[T]^{\ast}$ are given by the exponentials
$$e^{\lambda S} = \sum \lambda^k \frac{S^k}{k!}$$
(again, the factorial divisions are just notation, this works in any characteristic), so it suffices to show that these eigenvectors span a proper submodule. In turn it suffices to show that there exists an exponential power series $f$ which does not satisfy a linear homogeneous constant-coefficient differential equation $p \left( \frac{\partial}{\partial S} \right) f = 0$. This can be done with a cardinality argument but it's more fun to be explicit; over any field $F$ we can take a lacunary power series such as
$$f(S) = \sum_{n \ge 0} \frac{S^{n^2}}{(n^2)!}.$$
We can show that $f$ does not satisfy a differential equation of the form $p \left( \frac{\partial}{\partial S} \right) f = 0$ where $\deg p = d$ by looking at what happens once the terms of $f$ get farther apart than $d$; the recurrence is no longer able to cancel them, so $p \left( \frac{\partial}{\partial S} \right) f \neq 0$ from that point on.
Corollary: If $T$ has infinite spectrum or is not diagonalizable, then $T^{\ast}$ is not diagonalizable.