Taking the logarithm of:
$\displaystyle x=ay^2(by)^{\frac 1y}$
we obtain:
$$\log(x)=\log(a)+2\log(y)+{\log(by) \over y}$$
placing $z=\log(by)$:
$$\log(x)=\log(a)+2z+({\log(b)+z})e^{-z}$$
$$e^{-z}={\log(x/a)-2z \over {\log(b)+z}}$$
or:
$$e^{z}=\frac{\log(b)+z }{\log(x/a)-2z }$$
We know that mixed exponential/bilinear equations can be solved by the extended Lambert function $W_r(x)$, which can be represented by the following Lagrange inverting series:
$$z(A,t,s)=t- (t-s) \sum_{n=1} \frac{L_n' (n(t-s))}{n} e^{-nt} A^n$$
which is the solution of:
$$e^z=A\frac{z-t}{z-s}$$
Related questions
Related question concerning equations of the form:
$$e^z=A\frac{z-t}{z-s}$$
Lambert W function with rational polynomial
Solve $-B \ln y -A y \ln y + A y- A =0$ for $y$
Transcendental equations involving more than 2 terms
How to solve this equation for x? $0 = (x+k)e^{-(x+k)^2}+(x-k)e^{-(x-k)^2}$
References
On the generalization of the Lambert $W$ function with applications in theoretical physics, http://arxiv.org/abs/1408.3999
[68] C. E. Siewert and E. E. Burniston, "Solutions of the Equation $ze^z=a(z+b)$," Journal of Mathematical Analysis and Applications, 46 (1974) 329-337. http://www4.ncsu.edu/~ces/pdfversions/68.pdf