-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding
Video temporal grounding (VTG) is a fine-grained video understanding problem that aims to
ground relevant clips in untrimmed videos given natural language queries. Most existing
VTG models are built upon frame-wise final-layer CLIP features, aided by additional
temporal backbones (eg, SlowFast) with sophisticated temporal reasoning mechanisms. In
this work, we claim that CLIP itself already shows great potential for fine-grained spatial-
temporal modeling, as each layer offers distinct yet useful information under different …
ground relevant clips in untrimmed videos given natural language queries. Most existing
VTG models are built upon frame-wise final-layer CLIP features, aided by additional
temporal backbones (eg, SlowFast) with sophisticated temporal reasoning mechanisms. In
this work, we claim that CLIP itself already shows great potential for fine-grained spatial-
temporal modeling, as each layer offers distinct yet useful information under different …
Video temporal grounding (VTG) is a fine-grained video understanding problem that aims to ground relevant clips in untrimmed videos given natural language queries. Most existing VTG models are built upon frame-wise final-layer CLIP features, aided by additional temporal backbones (e.g., SlowFast) with sophisticated temporal reasoning mechanisms. In this work, we claim that CLIP itself already shows great potential for fine-grained spatial-temporal modeling, as each layer offers distinct yet useful information under different granularity levels. Motivated by this, we propose Reversed Recurrent Tuning (-Tuning), a parameter- and memory-efficient transfer learning framework for video temporal grounding. Our method learns a lightweight Block containing only 1.5% of the total parameters to perform progressive spatial-temporal modeling. Starting from the last layer of CLIP, Block recurrently aggregates spatial features from earlier layers, then refines temporal correlation conditioning on the given query, resulting in a coarse-to-fine scheme. -Tuning achieves state-of-the-art performance across three VTG tasks (i.e., moment retrieval, highlight detection, and video summarization) on six public benchmarks (i.e., QVHighlights, Charades-STA, Ego4D-NLQ, TACoS, YouTube Highlights, and TVSum) even without the additional backbone, demonstrating the significance and effectiveness of the proposed scheme. Our code is available at https://github.com/yeliudev/R2-Tuning.
arxiv.org
Showing the best result for this search. See all results