RM新时代有限公司

In the game "Halo," the

AI Cortana once said,

"I am his shield, I am his sword; I know him, along with his past and future."

Vision, as the foremost of the "six senses," accounts for over 70% of human external information absorption. If the long-term goal of AI is to create a robot with a brain, nerves, torso, and limbs, then the hardware foundation is the torso and limbs. More importantly, to enable the robot to see and think, hear and speak, we need to build the nerves and brain.

The goal of training visual models is to teach AI to see and understand the real world, where point cloud video understanding is crucial for agent-world interaction.

Recently, the top international computer vision conference CVPR 2023 was held in Vancouver, Canada. As one of the three top conferences in computer vision and pattern recognition, CVPR attracts significant attention. The paper by CloudWalk Technology and the joint research team, "PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos" (Masked Prediction-based Self-Supervised Learning for Point Cloud Videos) was successfully accepted.

01

Introduction

Great achievements have been made in parsing the real world from static point clouds. Recently, understanding point cloud videos has also gained increasing attention. Meanwhile, self-supervised learning can extract high-quality representations from unlabeled data, which will benefit the annotation-costly task of point cloud video understanding.
Therefore, we explore methods to learn representations from point cloud videos in a self-supervised manner. Although self-supervised learning paradigms based on contrastive learning and masked prediction have shown strong effectiveness in image and static point cloud domains, directly extending these methods to point cloud videos still presents many challenges.

In this paper, we propose PointCMP, a contrastive mask prediction framework for self-supervised learning on point cloud videos. PointCMP adopts a dual-branch structure to simultaneously learn local and global spatiotemporal information from point cloud videos. On top of this, we propose a mutual-similarity-based enhancement module for feature-based sample generation.
By computing the similarity between each local representation and the global semantics of the sample, we can identify key parts that are highly correlated with the semantics. Masking these key parts increases the challenge of the self-supervised prediction task, encouraging the model to learn more effective representations. Meanwhile, we also attempt to erase key feature channels to generate hard negative examples for global contrastive learning.

02

Method

Our PointCMP architecture is illustrated in Figure 1. Given a point cloud video, it is first evenly divided into multiple video clips. These clips are then fed into an encoder to obtain local tokens and a global token with global semantics. Next, they are passed to the mutual-similarity-based enhancement module.

Figure 1: Schematic diagram of PointCMP architecture

Intuitively, when local tokens with high similarity to the global token are visible, the prediction task becomes easier. Therefore, we mask these high-similarity local tokens to generate challenging masked samples. We select tokens with high similarity as key tokens. The point clouds covered by each local token usually overlap, while there is some information isolation between video clips.
Thus, we select the video clip containing the most key tokens and mask all local tokens aggregated from that clip. Additionally, we treat feature channels with high relevance as principal channels and erase them to generate hard negative samples. Intuitively, erasing these important principal component features will inevitably form a negative sample pair with the original sample.

We add the masked token sequence to positional encodings and feed it to a regressor to predict the representations at masked positions. The predicted representations form positive pairs with the corresponding original representations from the encoder, and negative pairs with the rest. We use the InfoNCE loss for this local contrastive learning branch.

Meanwhile, we also construct a global contrastive learning branch for the global representation of the sample. The token sequence completed by the regressor is passed through a pooling layer to obtain a new global representation, which forms a positive pair with the original global representation.
The token sequence after erasing principal feature channels is passed through a pooling layer to obtain global hard negative examples. Additionally, global representations of other videos in the same batch also serve as negative samples for the current sample. We also use the InfoNCE loss for this global contrastive learning branch.

03

Experiments

First, we fine-tune the pre-trained encoder to evaluate the representations learned by PointCMP. We use the MSRAction-3D dataset for both pre-training and end-to-end fine-tuning. When using PSTNet as the encoder, PointCMP pre-training brings significant accuracy improvements over the baseline.

Table 1: Validation results on the MSRAction-3D dataset

As shown in Table 1, when using 8 frames, action recognition accuracy improves from 83.50% to 89.56%. This indicates that PointCMP pre-training can learn beneficial knowledge from point cloud videos in a self-supervised manner, leading to higher accuracy after fine-tuning.

Then, we validate the effectiveness of the representations learned by PointCMP pre-training through linear experiments. Again, the MSRAction-3D dataset is used for pre-training and linear evaluation. The pre-trained encoder is frozen, and an additional linear classifier is added for supervised training. Our method surpasses the baseline under most frame settings. This demonstrates that PointCMP pre-training enables the model to learn high-quality representations.

Table 2: Validation results on the NTU-RGBD (Cross-Subject) dataset

Additionally, we conduct semi-supervised experiments on the NTU-RGBD dataset, where the model is fine-tuned with a portion of labeled data after self-supervised pre-training. From the results in Table 2, when using PSTNet as the encoder, fine-tuning with only 50% of labeled data can approach the fully supervised baseline accuracy. This indicates that PointCMP pre-training can mine knowledge inherent in unlabeled data, which not only saves labor costs but also further improves model performance by using the pre-trained model as initialization.

Figure 2: Visualization of high-similarity local tokens and their neighboring points (green)

In Figure 2, we further visualize the key local tokens with high similarity to the global token and their neighboring points. As we can see, points corresponding to key body parts involved in motion are highlighted. This is consistent with our intuition. By masking these key regions, the encoder is encouraged to utilize more context for mask prediction, thereby learning higher-quality representations.

Conclusion and Outlook

The advantage of self-supervised learning lies primarily in using auxiliary tasks to mine supervisory information from unlabeled data. Compared to training with task-specific labels as supervision, this not only saves annotation costs but also enables the model to learn more generalizable knowledge and representations valuable for various downstream tasks. In the data-driven era, this characteristic has led to a strong belief that self-supervised learning is the direction for artificial intelligence.

On the other hand, point cloud videos contain rich dynamic visual information, helping agents fully understand the real-time changing 3D world. Moreover, compared to traditional videos that rely mainly on texture information, point cloud videos provide more precise geometric information and spatial coordinates.
Therefore, point cloud videos can support tasks such as action recognition in low-visibility environments. Thus, point cloud video understanding is crucial for AI systems to interact with the world. Leveraging self-supervised techniques on massive data to advance point cloud video understanding may help us create an AI that can think and speak, listen and see.