Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Title: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
Authors: Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta
Published: 22nd December 2020 (Tuesday) @ 07:42:30
Link: http://arxiv.org/abs/2012.13255v1

Abstract

Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.

---

The basis of the LoRA paper

• An objective function’s intrinsic dimension measures the minimum number of parameters needed to reach satisfactory solutions to the respective objective - from Li et al., 2018: Measuring the Intrinsic Dimension of Objective Landscapes
• Practically: the intrinsic dimension represents the lowest dimensional subspace in which one can optimize the original objective function to within a certain level of approximation error.
• Computing the exact atrinsic dimensional of the objective function is computation intractable; therefore, we resort to heuristic methods to calculate an upper bound.
  • critique this assumes that your optimisation process is good enough to get you to that solution. If the optimisation we perform is inadequate, we cannot use this practical proxy to find the upper bound
• Li et al. 2018 proposed three actualizations of $P$, the projection from the smaller dimension to the model dimension $P:{\mathbb{R}}^d\to {\mathbb{R}}^D$
  • a random linear dense projection (${\theta }^{d}W$)
  • random linear sparse projection (${\theta }^d{W}_{sparse}$); and
  • random linear projection via the Fastfood transform (Le et al., 2013)