You're right to worry about your attempt, as Smiley1000 shows.
Instead, let's follow the hint. We're told that a point of $\mathscr{X}/X$ should be a pair $(e,x)$ where $e : \mathsf{Set} \to \mathscr{X}$ is a point of $\mathscr{X}$ and $x \in e^*(X)$. To intuitively see why, remember we can think of $\mathscr{X}/X$ as being a kind of "etale space" over $\mathscr{X}$. For simplicity let's picture a covering space $p : \mathscr{Y} \to \mathscr{X}$ -- then a point of $\mathscr{Y}$ is the data of a point in $\mathscr{X}$ and the choice of lift in the stalk $\mathscr{Y}_x = p^{-1}(x)$... I'll leave it as a nice exercise to see how this definition for spaces translates along the inclusion of spaces into topoi to become the hint.
So we want $(e,x)$ to somehow give us an adjunction between $\mathsf{Set}$ and $\mathscr{X}/X$... Let's do the inverse image first. Again, we think of an object $Y \in \mathscr{X}/X$ (really the topos $(\mathscr{X}/X)/Y$) as being an etale space over $\mathscr{X}/X$ whose fibre at a point $(e,x)$ is exactly $(e,x)^*(Y)$. But of course, $(\mathscr{X}/X)/Y \to \mathscr{X}/X \to \mathscr{X}$ means this is a covering space of $\mathscr{X}$ too, and we can relate these two perspectives! Indeed, some meditation on covering spaces shows that a point in the fibre at $(e,x)$ is just a point in the fibre over $e$ which maps to $x$. Again, translating this into the language of topoi we learn that the inverse image will be
$$
(e,x)^*(p: Y \to X) = \{ y \in e^*(Y) \mid e^*(p)(y) = x \}
$$
(using the fact that $e^*$ is a functor, so that $e^*(p) : e^*(Y) \to e^*(X)$). I'll leave it to you to check that this is cocontinuous and finitely continuous.
Now the question asks about the whole category of points, so we want to understand maps between these... Well, a map $\Phi : e \to f$ is just a natural transformation $e^* \to f^*$. We would like to build a natural transformation $(e_1,x_1)^* \to (e_2,x_2)^*$. So we want, for every $p : Y \to X$, a map from $(e_1,x_1)^*(p) \to (e_2,x_2)^*(p)$ which is natural in $p$. Explicitly, this is
$$
\{y_1 \in e_1^*(Y) \mid e_1^*(p)(y) = x_1 \} \to \{y_2 \in e_2^*(Y) \mid e_2^*(p)(y) = x_2 \}
$$
We have access to all our natural transformations $\Phi : e_1^* \to e_2^*$, and we know $\Phi_Y : e_1^*(Y) \to e_2^*(Y)$, so we're most of the way there! If moreoever $\Phi_X(x_1) = x_2$ then naturality of $\Phi$ will tell us that
$$e_2^*(p)(\Phi_Y(y_1)) = \Phi_X(e_1^*(p)(y_1)) = x_2$$
so that whenever $\Phi_X(x_1) = x_2$, $\Phi$ restricts to a natural transformation $(e_1,x_1)^* \to (e_2,x_2)^*$!
So where did we end? We learned that the points of $\mathscr{X}/X$ are pairs $(e,x)$ where $e$ is a point of $\mathscr{X}$ and $x \in e^*(X)$. Moreover, the inverse image part of $(e,x)$ sends an object $p : Y \to X$ to the set $\{y \in e^*(Y) \mid e^*(p)(y) = x \}$. The natural transformations $(e_1,x_1)^* \to (e_2,x_2)^*$ are exactly those natural transformations $\Phi : e_1^* \to e_2^*$ with the ~bonus property~ that $\Phi_X(x_1) = x_2$.
I hope this helps ^_^