Representation collapse

{{Short description|Phenomenon in machine learning}}

{{Short description|Phenomenon in machine learning}}

''Representation collapse'' is a [[phenomenon]] in [[machine learning]] and [[representation learning]] where a [[model]] maps different inputs to the same or very similar embeddings, which means it loses important information about how the data is spread out. It is frequently encountered in [[self-supervised learning]], especially within contrastive and non-contrastive frameworks, when training objectives or model architectures do not maintain variance across representations. Collapse results in degenerate solutions characterized by uninformative learned features, significantly impairing [[Downstream (networking)|downstream]] task performance. Various techniques have been proposed to mitigate representation collapse, including the use of negative samples, architectural asymmetry, stop-gradient operations, variance regularization, and redundancy reduction objectives, as seen in methods such as SimCLR, BYOL, and VICReg. Comprehending and averting representation collapse is regarded as a fundamental challenge in the advancement of stable and efficient self-supervised learning systems.{{cite journal |last1=Chen |first1=Ting |last2=Kornblith |first2=Simon |last3=Norouzi |first3=Mohammad |last4=Hinton |first4=Geoffrey |title=A Simple Framework for Contrastive Learning of Visual Representations |journal=Proceedings of the 37th International Conference on Machine Learning |year=2020}}{{cite journal |last1=Grill |first1=Jean-Bastien |last2=Strub |first2=Florian |last3=Altché |first3=Florent |last4=Tallec |first4=Corentin |title=Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning |journal=Advances in Neural Information Processing Systems |year=2020}}{{cite journal |last1=Bardes |first1=Adrien |last2=Ponce |first2=Jean |last3=LeCun |first3=Yann |title=VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning |journal=International Conference on Learning Representations |year=2022}}{{cite journal |last1=Zbontar |first1=Jure |last2=Jing |first2=Li |last3=Misra |first3=Ishan |last4=LeCun |first4=Yann |last5=Denis |first5=Stéphane |title=Barlow Twins: Self-Supervised Learning via Redundancy Reduction |journal=International Conference on Machine Learning |year=2021}}{{cite journal |last1=Jing |first1=Li |last2=Tian |first2=Yingli |title=Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |year=2021}}{{cite journal |last1=Wang |first1=Xiaolong |last2=Isola |first2=Phillip |title=Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere |journal=International Conference on Machine Learning |year=2020}}

'''Representation collapse''' is a [[phenomenon]] in [[machine learning]] and [[representation learning]] where a [[model]] maps different inputs to the same or very similar embeddings, which means it loses important information about how the data is spread out. It is frequently encountered in [[self-supervised learning]], especially within contrastive and non-contrastive frameworks, when training objectives or model architectures do not maintain variance across representations. Collapse results in degenerate solutions characterized by uninformative learned features, significantly impairing [[Downstream (networking)|downstream]] task performance. Various techniques have been proposed to mitigate representation collapse, including the use of negative samples, architectural asymmetry, stop-gradient operations, variance regularization, and redundancy reduction objectives, as seen in methods such as SimCLR, BYOL, and VICReg. Comprehending and averting representation collapse is regarded as a fundamental challenge in the advancement of stable and efficient self-supervised learning systems.{{cite journal |last1=Chen |first1=Ting |last2=Kornblith |first2=Simon |last3=Norouzi |first3=Mohammad |last4=Hinton |first4=Geoffrey |title=A Simple Framework for Contrastive Learning of Visual Representations |journal=Proceedings of the 37th International Conference on Machine Learning |year=2020}}{{cite journal |last1=Grill |first1=Jean-Bastien |last2=Strub |first2=Florian |last3=Altché |first3=Florent |last4=Tallec |first4=Corentin |title=Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning |journal=Advances in Neural Information Processing Systems |year=2020}}{{cite journal |last1=Bardes |first1=Adrien |last2=Ponce |first2=Jean |last3=LeCun |first3=Yann |title=VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning |journal=International Conference on Learning Representations |year=2022}}{{cite journal |last1=Zbontar |first1=Jure |last2=Jing |first2=Li |last3=Misra |first3=Ishan |last4=LeCun |first4=Yann |last5=Denis |first5=Stéphane |title=Barlow Twins: Self-Supervised Learning via Redundancy Reduction |journal=International Conference on Machine Learning |year=2021}}{{cite journal |last1=Jing |first1=Li |last2=Tian |first2=Yingli |title=Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |year=2021}}{{cite journal |last1=Wang |first1=Xiaolong |last2=Isola |first2=Phillip |title=Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere |journal=International Conference on Machine Learning |year=2020}}

== See also ==

== See also ==