Does using different optimizer change the loss landscape

Question

I plot the landscape using this code, and I notice the landscape shape has changed a lot. My understanding is that the optimizer does not change the loss landscape. But now I'm confused if its just that I made a mistake in generating the plot or that indeed the optimizer does change the landscape.

Answer 1

No, the optimizer does not change the loss landscape. The loss landscape consists of the possible parameters of your model and the incured loss of a model which has these parameters. We could for instance write it as $$ \lbrace (\theta , L(m_\theta (x), y) ) \mid \theta \in \Theta \rbrace, $$ where $\Theta$ is some set of admissible parameters, $m$ is your model, $L$ is the (average) loss, $x$ are features and $y$ are observations. An optimizer lets you find good parameters $\theta$, but does not change the landscape.

However, the package loss-landscapes you are referring to does not plot the loss landscape! It does two things (see the README.md):

• It plots a landscape based on a subspace of the parameter space $\Theta$, since the latter is too high-dimensional to be properly visualized.
• It plots this modified landscape 'around' the parameters $\theta^*$ of your fitted model.

Thus, there is no reason to believe your plot will look the same after changing the optimizer. In particular, if you use a different optimizer for fitting, you will most certainly estimate different parameters $\theta^*$ and so your plot will contain a different part of the (subspace of) the landscape.