We first cite the following theorem:[1]
Theorem. (Agnew, 1951) Let $f : (0, \infty) \to \mathbb{R}$ be locally integrable. Then the followings are equivalent:
(1) $\int_{0}^{\infty} \frac{f(at) - f(bt)}{t} \, \mathrm{d}t$ converges as an improper integral for each $\lambda = \log(a/b)$ in a set of positive measure.
(2) Both $\lim_{x \to \infty} \frac{1}{x}\int_{1}^{x} f(t) \, \mathrm{d}t$ and $\lim_{x \to 0^+} x \int_{x}^{1} \frac{f(t)}{t^2} \, \mathrm{d}t$ converge.
(3) Both $\lim_{x \to \infty} \frac{1}{x}\int_{0}^{x} f(t) \, \mathrm{d}t$ and $\lim_{x \to 0^+} \frac{1}{x}\int_{0}^{x} f(t) \, \mathrm{d}t$ converge.
Moreover, in such case, we have
\begin{align*}
C_{\infty} &: = \lim_{x \to \infty} \frac{1}{x}\int_{0}^{x} f(t) \, \mathrm{d}t = \lim_{x \to \infty} \frac{1}{x}\int_{1}^{x} f(t) \, \mathrm{d}t, \\
C_{0} &: = \lim_{x \to 0^+} \frac{1}{x}\int_{0}^{x} f(t) \, \mathrm{d}t = \lim_{x \to 0^+} x \int_{x}^{1} \frac{f(t)}{t^2} \, \mathrm{d}t
\end{align*}
ad well as
$$\int_{0}^{\infty} \frac{f(at) - f(bt)}{t} \, \mathrm{d}t = \lambda (C_{\infty} - C_0)$$
for each pair of positive numbers $a$ and $b$.
So the convergence of the standard Frullani's integral is intimately related to the Cesàro means near $0$ and $\infty$. The proof essentially hinges on the property of the Haar measure $x^{-1} \, \mathrm{d}x$ on the multiplicative group $(0, \infty)$. That being said, it seems unlikely to me that there exists a useful necessary condition for the existence of OP's generalized Frullani's integral.
Still, we may try to produce some sufficient conditions: To make the argument simple, we impose the following assumptions:
$f \in C([0,\infty])$, meaning that that $f$ is continuous on $[0, \infty)$ and $f(\infty):=\lim_{x\to\infty}f(x)$ converges.
$g : [0, \infty) \to [0, \infty)$ is a strictly increasing, continuous bijection with the inverse $h = g^{-1}$.
Then for any $a, b > 0$ and $0 < r < R$,
\begin{align*}
&\int_{r}^{R} \frac{f(ag(x)) - f(bg(x))}{x} \, \mathrm{d}x \\
&= \int_{r}^{R} \frac{f(ag(x))}{x} \, \mathrm{d}x - \int_{r}^{R} \frac{f(bg(x))}{x} \, \mathrm{d}x \\
&= \int_{ag(r)}^{ag(R)} f(u) \, \mathrm{d}\log h(u/a) - \int_{bg(r)}^{bg(R)} f(u) \, \mathrm{d}\log h(u/b) \\
&= \int_{ag(r)}^{bg(r)} f(u) \, \mathrm{d}\log h(u/a) - \int_{ag(R)}^{bg(R)} f(u) \, \mathrm{d}\log h(u/b) \\
&\quad + \int_{bg(r)}^{ag(R)} f(u) \, \mathrm{d}\log \left( \frac{h(u/a)}{h(u/b)} \right).
\end{align*}
Using the continuity of $f$, we may provide a sufficient condition for which the above expression converges as $r \to 0^+$ and $R \to \infty$ for arbitrary $a, b > 0$:
Condition. For any $c > 0$, each of the following converges:
$$ \lim_{r \to 0^+} \frac{h(cr)}{h(r)}, \qquad \lim_{R \to 0^+} \frac{h(cR)}{h(R)}, \qquad \int_{0}^{\infty} \left| \mathrm{d}\log\left(\frac{h(cu)}{h(u)} \right) \right| $$
Indeed, if this condition holds, then Karamata's characterization theorem for regularly varying functions tells that there exist $p, q \geq 0$ satisfying
$$ \lim_{r \to 0^+} \frac{h(cr)}{h(r)} = c^{p} \qquad \text{and} \qquad \lim_{R \to 0^+} \frac{h(cR)}{h(R)} = c^{q} $$
for all $c > 0$. Then it is not hard to prove that
\begin{align*}
\int_{0}^{\infty} \frac{f(ag(x)) - f(bg(x))}{x} \, \mathrm{d}x
&= (q f(\infty) - p f(0)) \log ( a/b ) + \int_{0}^{\infty} f(u) \, \mathrm{d}\log \left( \frac{h(u/a)}{h(u/b)} \right).
\end{align*}
Example 1. If $g(x) = x^{d}$ for some $d > 0$, then $p = q = \frac{1}{d}$ and $\mathrm{d}\log \left( \frac{h(u/a)}{h(u/b)} \right) = 0$. So we get
\begin{align*}
\int_{0}^{\infty} \frac{f(ax^{d}) - f(bx^{d})}{x} \, \mathrm{d}x
&= \frac{(f(\infty) - f(0)) \log ( a/b )}{d}.
\end{align*}
This can also be proved directly from the standard Frullani's integral applied to $x \mapsto f(x^d)$.
Example 2. If $g(x) = \frac{x+\sqrt{x^2+4x}}{2}$, then $h(u) = \frac{u^2}{u+1}$, and so, $p = 2$ and $q = 1$. So we get
\begin{align*}
\int_{0}^{\infty} \frac{f(ag(x)) - f(bg(x))}{x} \, \mathrm{d}x
&= (f(\infty) - 2f(0)) \log ( a/b ) + \int_{0}^{\infty} \frac{(a-b)f(u)}{(u+a)(u+b)} \, \mathrm{d}u.
\end{align*}
As a sanity check, plugging $f \equiv 1$ shows that both sides are zero.
References.
- Agnew, Ralph Palmer. "Mean Values and Frullani Integrals." Proceedings of the American Mathematical Society 2, no. 2 (1951): 237-41. Accessed June 22, 2020. doi:10.2307/2032493.
