C \left( \frac{\sqrt{k}}{\hat{u}_t ^2} + \frac{1}{\hat{u}_t} \right),$$ where $\hat{u}_t \geq  (1 - t) \lambda_{\min}(\Sigma_{S}) + t \lambda_{\min}(\hat{\Sigma}_{S})$. By in and Weyl’s theorem, and using the fact that $ u > u_n$, on an event with probability at lest $1 - \frac{1}{n}$, $$\left\| G\left( 1-t)\psi_{S} + t \hat
 \psi_{S} \right) \right\| \leq C \left( \frac{\sqrt{k}}{u_n ^2} + \frac{1}{u_n} \right) \leq C \frac{\sqrt{k}}{u_n^2},$$ where in the last inequality we assume $n$ large enough so that $k \geq u_n^2$. The previous bound does not depend on $t$, $j$ or $P$. The result now follows. $\Box$

Appendix 2: Proofs of the results in 
=====================================

In all the proofs of the results from , we will condition on the outcome of the sample splitting step, resulting in the random equipartition $\mathcal{I}_{1,n}$ and $\mathcal{I}_{2,n}$ of $\{1,\ldots,2n\}$, and on $\mathcal{D}_{1,n}$. Thus, we can treat the outcome of the model selection and estimation procedure $w_n$ on $\mathcal{D}_{1,n}$ as a fixed. As a result, we regard ${\widehat{S}}$ as a deterministic, non-empty subset of $\{1,\ldots,d\}$ of size by $k < d$ and the projection parameter $\beta_{{\widehat{S}}}$ as a fixed vector of length $k$. Similarly, for the LOCO parameter $\gamma_{{\widehat{S}}}$, the quantities $\widehat{\beta}_{{\widehat{S}}}$ and $\widehat{\beta}_{{\widehat{S}}(j)}$, for $j \in {\widehat{S}}$, which depend on $\mathcal{D}_{1,n}$ also become fixed. Due to the independence of $\mathcal{D}_{1,n}$ and $\mathcal{D}_{2,n}$, all the probabilistic statements made in the proofs are therefore referring to the randomness in $\mathcal{D}_{2,n}$ only. Since all our bounds will depend on $\mathcal{D}_{1,n}$ through the cardinality of ${\widehat{S}}$, which is fixed at $k$, the same bounds will hold uniformly over all possible values taken on by $\mathcal{D}_{1,n}$ and $\mathcal{I}_{1,n}$ and all possible outcomes of all model selection and estimation procedures $w_n \in \mathcal{W}_n$ run on $\mathcal{D}_{1,n}$. In