Visual Prompt Tuning

Title: Visual Prompt Tuning
Authors: Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim
Published: 23rd March 2022 (Wednesday) @ 01:17:16
Link: http://arxiv.org/abs/2203.12119v2

Abstract

The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.

---

Very much builds on top of Prefix-Tuning Optimizing Continuous Prompts for Generation (arXived in Jan. 21; this paper in March ‘22)

Visual-Prompt Tuning (VPT; §3.2; p. 5)

There are two methods:

1. VPT-Shallow - first Transformer layer only
2. VPT-Deep - prompts introduced at every Transformer layer

Quoting section from paper:

Given a pre-trained Transformer model, we introduce a set of $p$ continuous embeddings of dimension $d$, i.e., prompts, in the input space after the Embed layer. Only the task-specific prompts are being updated during fine-tuning, while the Transformer backbone is kept frozen. Depending on the number of Transformer layers involved, our approach has two variants, VPT-sHALLOW and VPT-DEEP, as shown in Fig. 2.

VPT-Shallow. Prompts are inserted into the first Transformer layer $L_1$ only. Each prompt token is a learnable $d$-dimensional vector. A collection of $p$ prompts is denoted as $\mathbf{P}=\left\{{\mathbf{p}}^k\in {\mathbb{R}}^d\mid k\in \mathbb{N},1\le k\le p\right\}$, the shallow-prompted ViT is:

$$\begin{array}{rl}\left[{\mathbf{x}}_1,{\mathbf{Z}}_1,{\mathbf{E}}_1\right]& =L_1\left(\left[{\mathbf{x}}_0,\mathbf{P},{\mathbf{E}}_0\right]\right)\\ \left[{\mathbf{x}}_i,{\mathbf{Z}}_i,{\mathbf{E}}_i\right]& =L_i\left(\left[{\mathbf{x}}_{i-1},{\mathbf{Z}}_{i-1},{\mathbf{E}}_{i-1}\right]\right)\\ \mathbf{y}& =\mathrm{Head}\left({\mathbf{x}}_N\right),\end{array}i=2,3,\dots ,N$$

where ${\mathbf{Z}}_i\in {\mathbb{R}}^{p\times d}$ represents the features computed by the $i$-th Transformer layer, and $\left[{\mathbf{x}}_i,{\mathbf{Z}}_i,{\mathbf{E}}_i\right]\in {\mathbb{R}}^{(1+p+m)\times d}$. The colors $\bullet$ and $\bullet$ indicate learnable and frozen parameters, respectively. Notably for ViT, ${\mathbf{x}}_N$ is invariant to the location of prompts since they are inserted after positional encoding, e.g., $\left[{\mathbf{x}}_0,\mathbf{P},{\mathbf{E}}_0\right]$ and $\left[{\mathbf{x}}_0,{\mathbf{E}}_0,\mathbf{P}\right]$ are mathematically equivalent. This also applies to VPT-Deep.

VPT-Deep. Prompts are introduced at every Transformer layer’s input space. For ( $i+1$ )-th Layer $L_{i+1}$, we denote the collection of input learnable prompts as ${\mathbf{P}}_i=\left\{{\mathbf{p}}_i^k\in {\mathbb{R}}^d\mid k\in \mathbb{N},1\le k\le m\right\}$. The deep-prompted ViT is formulated as:

$$\begin{array}{rl}\left[{\mathbf{x}}_i,\dots ,{\mathbf{E}}_i\right]& =L_i\left(\left[{\mathbf{x}}_{i-1},{\mathbf{P}}_{i-1},{\mathbf{E}}_{i-1}\right]\right)i=1,2,\dots ,N\\ \mathbf{y}& =\mathrm{Head}\left({\mathbf{x}}_N\right)\end{array}$$

Storing Visual Prompts. VPT is beneficial in presence of multiple downstream tasks. We only need to store the learned prompts and classification head for each task and re-use the original copy of the pre-trained Transformer model, significantly reducing the storage cost. For instance, given a ViT-Base with 86 million (M) parameters and $d=768,50$ shallow prompts and deep prompts yield additional $p\times d=50\times 768=0.038\mathrm{M}$, and $N\times p\times d=0.46\mathrm{M}$ parameters, amounting to only $0.04\%$ and $0.53\%$ of all ViT-Base parameters, respectively.