Here is an idea I came up but I was only able essentially reduce it to this question. I had hoped to prove that the mapping torus of a diffeomorphism of an orientable $3$-manifold is zero using the fact that orientable $3$-manifolds have trivial tangent bundles and the definition of Stiefel-Whitney classes as obstruction classes. In the end this gave a good method to produce counterexamples I think.
Let $M$ denote an orientable $3$-manifold and $f$ and orientation preserving diffeomorphism. I will denote the mapping Torus by $T_f$ and the inclusion of some fiber by $\iota\colon M\to T_f$. Fix a trivialization of $T\iota(M)$, which is possible by the aforementioned fact. Now we know that $\iota^*(w_i(S_f))=w_i(\iota^*TS_f)=w_i(TM\oplus \mathbb{R})=0$ we conclude that $w_i(S_f)$ comes from some class in $H^i(S_f,\iota(M);\mathbb{Z}/2\mathbb{Z})$. Using excision the inclusion of the pair $(M\times I,M \times \partial I)\to (S_f,\iota(M))$ induces an isomorphism on cohomology. Therefore we have to understand how the Stiefel-Whitney classes of $(M\times I,\partial M\times I)$ behave.
Since $T(M\times I)\cong \pi^* TM\oplus \mathbb{R}$, where $\pi$ denotes the projection $M\times I\to M$ and this splitting respects the fixed framing at $M\times \partial I$, we have to understand $w_3(\pi^*TM,\pi^*TM|_{M\times \partial I})$. Note that this is the mod $2$ reduction of the relative Euler class. Furthermore note that if we fix some non-vanishing section $\phi$ of $\pi^*TM|_{M\times \{0\}}$ then the section at $\pi^*TM|_{M\times \{1\}}$ is given by $f_* \phi((x,1))=Df_{f^{-1}(x)}(\phi(f^{-1}(x))$. All in all this should imply that $w_3$ is the mod $2$ reduction of the obstruction class for a homotopy between $\phi$ and $f_*(\phi)$.
Therefore we are left with the question how do homotopy classes of non-vanishing vector fields on orientable $3$-manifolds behave under diffeomorphisms of said manifold, which is exactly the aforementioned question. Nevertheless note that $[M,S^2]$, which is the set of homotopy classes of vector fields, surjects quite naturally to $H^2(M;\mathbb{Z})$. So maybe it is possible to deduce the existence of a vector field $\phi$ and a diffeomorphism $f$ such that the obstruction class for a homtopy between $f_* \phi$ and $\phi$ is non-zero mod $2$ using the action of $f$ on $H^2(M)$, but I'm tired right now so I will think about this last part tomorrow.