SHIFT

SHIFT

Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning

Restore dynamic degree after fine-tuning — with pixel-motion rewards and a sampler-agnostic hybrid RL framework.

Paper PDF arXiv BibTeX Code
ECCV 2026

Xi Ye1,2,*, Wenjia Yang1,*, Yangyang Xu1,2, Xiaoyang Liu1,3, Duo Su1, Mengfei Xia2, Jun Zhu1

1Tsinghua University  ·  2Ant Group  ·  3University of Chinese Academy of Sciences  ·  *Equal contribution

Abstract

Image-conditioned video diffusion models achieve impressive visual realism but often suffer from weakened motion fidelity — reduced dynamics or degraded long-term temporal coherence — especially after fine-tuning. We introduce pixel-motion rewards based on pixel flux dynamics, capturing both instantaneous and long-term motion consistency. We further propose Smooth Hybrid Fine-Tuning (SHIFT), a scalable reward-driven framework that unifies supervised fine-tuning and advantage-weighted fine-tuning. With adversarial advantages, SHIFT improves convergence and mitigates reward hacking, efficiently resolving dynamic-degree collapse in modern video diffusion supervised fine-tuning.

Method

Pixel-motion reward models: Instantaneous Motion Reward and Long-term Motion Reward
(a) Pixel-motion rewards: Instantaneous Motion Reward (IMR) from optical-flow residuals, and Long-term Motion Reward (LMR) from point trajectories.
SHIFT Smooth Hybrid Fine-Tuning framework diagram
(b) SHIFT fuses online advantage-weighted updates on model rollouts with an offline SFT anchor on real data, while adversarially refreshing the reward model.

Pixel-motion rewards

IMR and LMR supervise local coherence and sustained dynamics from optical flow and point tracks — no human labels.

Hybrid forward fine-tuning

Advantage-weighted regression on rollouts plus an SFT data anchor. Sampler-agnostic; no reverse-trajectory storage.

Adversarial advantages

Co-train the reward discriminator online to track distribution shift and reduce reward hacking.

Results

On SVD and Wan2.2, SHIFT improves motion fidelity, preserves appearance, and trains nearly 8× faster than FlowGRPO.

86.70 VBench Motion (SVD) · best
84.69 VBench Overall (SVD) · best
2.46× Training time vs SFT · vs 19.35× FlowGRPO
86.40 VBench Overall (Wan2.2) · best
Method Appearance ↑ Motion ↑ Overall ↑ Motion Score ↑ FVD ↓ Time ↓
Base 83.68 85.89 84.41 4.39 395.87 —
SFT 85.07 76.09 82.08 2.69 322.81 1×
Track4Gen 85.01 76.44 82.15 2.84 316.91 1.07×
DenseDPO 83.73 86.03 84.50 4.34 399.73 2.40×
FlowGRPO 83.61 86.67 84.63 4.44 396.56 19.35×
SHIFT (Ours) 83.69 86.70 84.69 4.40 396.45 2.46×

VBench-I2V on SVD. SFT / Track4Gen improve FVD but collapse motion; SHIFT keeps appearance and lifts motion efficiently.

BibTeX

If you find this work useful, please cite:

@inproceedings{ye2026shift,
  title     = {SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning},
  author    = {Ye, Xi and Yang, Wenjia and Xu, Yangyang and Liu, Xiaoyang and Su, Duo and Xia, Mengfei and Zhu, Jun},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}