http://www.ee.columbia.edu/ln/dvmm/publications/17/zhang2017visual.pdf
Visual Translation Embedding Network for Visual Relation Detection Hanwang Zhang† , Zawlin Kyaw‡ , Shih-Fu Chang† , Tat-Seng Chua‡ †Columbia University, ‡National University of Singapore
亮点
- 视觉关系预测问题的分析与化简：把一种视觉关系理解为在特征空间从主语到宾语的一种变换，很有效、很直白
- 实验设计的很棒，从多角度进行了分析对比：语言空间划分，多任务对物体检测的提升，零次学习等。
现有工作
- Mature visual detection [16, 35]
- Burgeoning visual captioning and question answering [2, 4]
- directly bridge the visual model (e.g., CNN) and the language model (e.g., RNN), but fall short in modeling and understanding the relationships between objects.
- poor generalization ability
- Visual Relation Detection: a visual relation as a subject-predicate-object triplet
- joint models, a relation triplet is considered as a unique class [3, 9, 33, 37].
- the long-tailed distribution is an inherent defect for scalability.
- separate model
- modeling the large visual variance of predicates is challenging.
- language priors to boost relation detection
主要思想
Translation Embedding 视觉关系预测的难点主要是：对于N个物体和R种谓语，有N^2R种关系，是一个组合爆炸问题。解决这个问题常用的办法是：
- 估计谓语，不估计关系，缺点：对于不同的主语、宾语，图像视觉差异巨大
受Translation Embedding (TransE) 启发，文章中将视觉关系看作在特征空间上从主语到宾语的一种映射，在低维空间上关系元组可看作向量变换，例如person+ride ≈ bike.
Knowledge Transfer in Relation 物体的识别和谓语的识别是互惠的。通过使用类别名、位置、视觉特征三种特征和端对端训练网络，使物体和谓语之前的隐含关系在网络中能够学习到。
算法
Visual Translation Embedding
Loss function
Feature Extraction Layer
classname + location + visual feature 不同的特征对不同的谓语（动词、介词、空间位置、对比）都有不一样的作用
Bilinear Interpolation
In order to achieve object-relation knowledge transfer, the relation error should be back-propagated to the object detection network and thus refines the objects. We replace the RoI pooling layer with bilinear interpolation [18]. It is a smooth function of two inputs:
结果
Translation embeding: ＋18%
object detection ＋0.6% ～ 0.3%
State-of-art:
- Phrase Det. +3% ~ 6%
- Relation Det. +1%
- Retrieval -1% ~ 2%
- Zero-shot phrase detection
- Phrase Det. －0.7% (without language prior)
- Relation Det. －1.4%
- Retrieval ＋0.2％
问题
- 两个物体之间可能有多种关系，比如person ride elephant，同时也存在person short elephant但是文章中的方法无法表示多样化的关系
- 没有使用语言先验知识，使用多模态信息可能会有所提升