Let $a,b$ be nonzero and different real numbers. Let $f: [0,1] \to R $ with
$$ f(x) = \left\{ \begin{array}{lr} ax & : x \in \mathbb{Q}\\ bx & : x \notin \mathbb{Q} \end{array} \right. $$
Let $D$ be the set of discontinuities of $f$. I want to find out what $D$ is but I am having difficulties. Do I have to consider cases on $a$ and $b$ ?
Break into two cases, the case where $a=b$ and the case where $a\neq b$. When $a\neq b$ visualize the function as two lines (with holes in them) intersecting at the origin. Remember that every irrational point is a limit point of rationals and every irrational point is a limit point of rationals.
For all $x_0\in [0,1]$, and if $f$ is continous at $x_0$, then it needs two conditions, 1)$\lim_{x\to x_o}f(x)$ must exists 2)$\lim_{x\to x_o}f(x)=f(x_0)$.
First we check whether $\lim_{x\to x_o}f(x)$ exists, this is equivalent to $$\lim_{x\to x_o, x\in \mathbb{Q}}f(x)=\lim_{x\to x_o, x\in \mathbb{R}\setminus\mathbb{Q}}f(x)$$ Hence we must have $ax_0=bx_0\iff (a-b)x_0=0$.
If $a\neq b$, this can only be satisfied by $x_0=0$. Hence $[0,1]\setminus\{0\}\subset D$. Then we check whether it's continuous at $0$. Indeed it is, since $\lim_{x\to x_o, x\in \mathbb{Q}}f(x)=\lim_{x\to x_o, x\in \mathbb{R}\setminus\mathbb{Q}}f(x)=0=f(0)$. So the function is continuous at $0$ and $D=[0,1]\setminus\{0\}$.
If $a=b$, then $f(x)=ax \quad x\in [0,1]$. So $f$ is continous at $[0,1]$, $D=\emptyset$.
Peace be upon you
The answer is $(0,1]$, because the two lines diverges by passing from 0. The discontinuity can be written mathematically as you write the discontinuity of Dirichlet function.
It is a modification of the Dirichlet function see : http://en.wikipedia.org/wiki/Nowhere_continuous_function if $a\neq b$ is discontinuous at every point, except at the origin, where it is continuous.
