arxivst stuff from arxiv that you should probably bookmark

Time-Contrastive Networks: Self-Supervised Learning from Multi-View Observation

Abstract · Apr 23, 2017 06:03 ·

pouring container viewpoint videos self contrastive supervised viewpoints imitation attributes cs-cv cs-ro

Arxiv Abstract

  • Pierre Sermanet
  • Corey Lynch
  • Jasmine Hsu
  • Sergey Levine

We propose a self-supervised approach for learning representations entirely from unlabeled videos recorded from multiple viewpoints. This is particularly relevant to robotic imitation learning, which requires a viewpoint-invariant understanding of the relationships between humans and their environment, including object interactions, attributes and body pose. We train our representations using a triplet loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. This signal encourages our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. Our experiments demonstrate that such a representation even acquires some degree of invariance to object instance. We demonstrate that our model can correctly identify corresponding steps in complex object interactions, such as pouring, across different videos with different instances. We also show what is, to the best of our knowledge, the first self-supervised results for end-to-end imitation learning of human motions by a real robot.

Read the paper (pdf) »