Bayesian optimisation in deeplearning

Question

Has anyone tried using Bayesian optimisation to get best learning rates, and other hyperparameters for deeplearning.

How to change the parameters between the training. Any examples on callbacks?

Can I see some codes to implement them?

Answer 1

Currently I am fiddling around with Ax/BoTorch to optimize a ResNet50 for Object classification (and later on detection via SSD). I found it quite hard to apply the samples from the docs to my own model, so this is a very first prototype. Just have a look at this GIST...

https://gist.github.com/pinkerltm/74f98485d1408849476fb02686d0c228

So far I can tell, that it is somehow optimizing the learning rate into the right direction, but I think I have to wire some more Hyperparameters properly and perhaps optimize longer to get a result which is compareable to simply defining the best parameters manually so far, as Accuracy with the optimized Learning rate only gets around 67% after several 100 epochs while at least 71% should be easily possible with this model.

I am also very interested in sharing experiences and findings on this. I will edit my answer here accordingly so that it should provide at least an simple example of optimizing a CNN Classification Network with Ax/BoTorch.

Answer 2

I haven't had much experience with deep learning, but I have tried it on "vanilla" ML techniques.

In Python, I've had some success using a package called hyperopt (link), although if you're interested in Bayesian methods then take a look at Spearmint (link).

If you're using R, then take a look at rBayesianOptimization.