arXiv Analytics

Sign in

arXiv:2009.07265 [cs.CV]AbstractReferencesReviewsResources

Understanding Deformable Alignment in Video Super-Resolution

Kelvin C. K. Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy

Published 2020-09-15Version 1

Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.

Related articles: Most relevant | Search more
arXiv:1803.05268 [cs.CV] (Published 2018-03-14)
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
arXiv:1903.04772 [cs.CV] (Published 2019-03-12)
Paradox in Deep Neural Networks: Similar yet Different while Different yet Similar
arXiv:1301.0435 [cs.CV] (Published 2013-01-03)
Investigating the performance of Correspondence Algorithms in Vision based Driver-assistance in Indoor Environment