-

We highlight the fact that the limiting SDE of a discrete process, $$\label{e:disc-mcmc-new}
    w_{k+1} = w_k - s\nabla U(w_k) + \sqrt{s} \xi(w_k, \eta_k),$$ depends only on the covariance matrix of $\xi$. More specifically, as long as $\xi$ satisfies $\sqrt{\E{\xi(w, \eta)\xi(w, \eta)^T}} = M(w)$, will have as its limiting SDE, *regardless of higher moments of $\xi$*. This fact, combined with Theorem \[t:main\_nongaussian\], means that in the limit of $\delta \to 0$ and $k\to \infty$, the distribution of $w_k$ will be determined by the covariance of $\xi$ alone. An immediate consequence is the following: *at convergence, the test performance of any Langevin MCMC-like algorithm is almost entirely determined by the covariance of its noise term.*

Returning to the case of SGD algorithms, since the noise covariance is $M(x)^2 = \frac{\delta}{b} H(x)$ (see ), we know that the ratio of step size $\delta$ to batch size $b$ is an important quantity which can dictate the test error of the algorithm; this observation has been made many times in prior work [@jastrzkebski2017three; @he2019control], and our results in this paper are in line with these observations. Here, we move one step further, and provide experimental evidence to show that more fundamentally, it is the noise covariance in the constant-noise limit that controls the test error.

To verify this empirically, we propose the following algorithm called *large-noise SGD*.

\[def:large\_noise\_sgd\] An $(s, \sigma, b_1, b_2)$-large-noise SGD is an algorithm that aims to minimize  using the following updates: $$\begin{aligned}
\numberthis\label{e:sgd-noisy}
    & w_{k+1} = w_k - \frac{s}{b_1} \sum_{i \in \eta_k} \nabla U_i(w_k) \\
    & \qquad + \frac{\sigma \sqrt{s}}{b_2} \left( \sum_{i\in \eta_k'}\nabla U_i(w_k) - \sum_{i\in\eta_k''}\nabla U_i(w_k) \right),\end{aligned}$$ where $\eta_k$, $\eta_k'$, and $\eta_k''$ are minibatches of sizes $b_1$, $b_2$, and $b_2$, sampled uniformly at random from $\{1, \ldots, n\}$ with replacement. The three minibatches are 