Learning Action Changes by Measuring Verb-Adverb Textual Relationships

Title: Learning Action Changes by Measuring Verb-Adverb Textual Relationships
Authors: Davide Moltisanti, Frank Keller, Hakan Bilen, Laura Sevilla-Lara
Published: 27th March 2023 (Monday) @ 10:53:38
Link: http://arxiv.org/abs/2303.15086v2

Abstract

The goal of this work is to understand the way actions are performed in videos. That is, given a video, we aim to predict an adverb indicating a modification applied to the action (e.g. cut “finely”). We cast this problem as a regression task. We measure textual relationships between verbs and adverbs to generate a regression target representing the action change we aim to learn. We test our approach on a range of datasets and achieve state-of-the-art results on both adverb prediction and antonym classification. Furthermore, we outperform previous work when we lift two commonly assumed conditions: the availability of action labels during testing and the pairing of adverbs as antonyms. Existing datasets for adverb recognition are either noisy, which makes learning difficult, or contain actions whose appearance is not influenced by adverbs, which makes evaluation less reliable. To address this, we collect a new high quality dataset: Adverbs in Recipes (AIR). We focus on instructional recipes videos, curating a set of actions that exhibit meaningful visual changes when performed differently. Videos in AIR are more tightly trimmed and were manually reviewed by multiple annotators to ensure high labelling quality. Results show that models learn better from AIR given its cleaner videos. At the same time, adverb prediction on AIR is challenging, demonstrating that there is considerable room for improvement.

---

Quick Notes

• Task: “adverb recognition” - potential useful applications in robotics and retrieval
• Adverbs harder to classify than e.g. events
• Polysemy makes adverb prediction hard e.g. “coarsely grind” vs “coarsely paint” (weird second example)
• The state-of-theart approach [6] treats adverbs as learnable parameters that modify actions.
  • Specifically, the action change is learnt during training by contrasting antonyms, i.e. opposite adverbs.
• We show that learning adverbs in such manner can be difficult and can limit the ability of the model to generalise.
• Propose to define action changes by measuring distances in a text embedding space, and aim to learn such change from the video through regression.
• Approach achieves new state-of-the-art results through extensive experiments.
• We also lift two major assumptions made in previous work [6, 7]:
  • the availability of action labels during testing
  • the pairing of opposite adverbs as antonyms.
• Our method achieves stronger performance especially when the above assumptions are relaxed
• Crucial contribution is the dataset they build: Adverbs in Recipes: the AIR Dataset

Method

Objective: They use a distance between adverbs and their antonyms and scale this by the cosine similarity between adverbs and their respective verbs in order to make the objective meaningful.

• $\delta (v,a)$ is this metric 👆
• $h(x)$ indicates taking the antonym of adverb $x$

$$Y^R(v)=\left(t_i,\forall i\in \{1,2,\dots ,A\}\right)=\{\begin{array}{cl}\delta (v,a)& \text{ if }{y}_i=a\\ -\delta (v,a)& \text{ if }{y}_i=h(a)\\ 0& \text{ otherwise }\end{array}$$

Table 2. Results obtained using the action label during inference.

Table 2. Results obtained using the action label during inference. mAP W/M: mean Average Precision with weighted (W) and macro (M) averaging. Acc-A: adverb-vs-antonym accuracy. Coloured rows indicate variants of our method. Bold denotes best result per column. In instructional datasets (left) adverbs are action-focused, so these are more reliable benchmarks to learn action changes. In captioning datasets (right) adverbs are more descriptive and do not influence the action significantly. As such, these datasets are less reliable.