Everything about Representation collapse
Phenomenon in machine learning
Representation collapse is a phenomenon in machine learning and representation learning where a model maps different inputs to the same or very similar embeddings, which means it loses important information about how the data is spread out. It is frequently encountered in self-supervised learning, especially within contrastive and non-contrastive frameworks, when training objectives or model architectures do not maintain variance across representations. Collapse results in degenerate solutions characterized by uninformative learned features, significantly impairing downstream task performance. Various techniques have been proposed to mitigate representation collapse, including the use of negative samples, architectural asymmetry, stop-gradient operations, variance regularization, and redundancy reduction objectives, as seen in methods such as SimCLR, BYOL, and VICReg. Comprehending and averting representation collapse is regarded as a fundamental challenge in the advancement of stable and efficient self-supervised learning systems. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Representation collapseA 5-minute orientation
- Skim a recent paper: Latent Space Collapse in Materials Representation Learning: A Systems-Level Conceptual AnalysisInfluential work since 2022
- Pick up a classic book: Red MoonThe most-reprinted book on the subject
- Read the landmark paper: Single-force representation of earthquakes due to landslides or the collapse of cavernsCited 38 times
- Go sideways: Self-supervised learningA closely related topic
- Go sideways: Representation learningA closely related topic
📖 Definitions
No dictionary entry for this exact phrase.
🗂️ The subject at a glance
No outline available.
🏆 Landmark papers (most cited)
Single-force representation of earthquakes due to landslides or the collapse of caverns
Collapse and reorganization patterns of social knowledge representation in evolving semantic networks
A progressive method for the collapse of river representation considering geographical characteristics
COLLAPSE : A representation learning framework for identification and characterization of protein structural sites
Auto-Encoding Generative Adversarial Networks towards Mode Collapse Reduction and Feature Representation Enhancement
PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse
Representation of connection behaviour for progressive collapse response
A Taxonomy and Theoretical Analysis of Collapse Phenomena in Unsupervised Representation Learning
🆕 Influential since 2022
Latent Space Collapse in Materials Representation Learning: A Systems-Level Conceptual Analysis
Representation and Geometric Collapse in Spatiotemporal EEG Classifiers: A Mathematical Diagnostic Framework
Addressing class collision and representation collapse in contrastive clustering via pseudo-supervised triplet and InfoNCE losses
When contexts collapse: How ubiquitous video cameras in the home during COVID-19 pandemic lockdowns transformed family representation
📚 Books worth reading
Red MoonKim Stanley Robinson, Maxwell Hamilton · 2018
Time and RealismYuval Dolev · 2007
The Collapse of the Spanish Republic, 1933-1936Stanley G. Payne · 2005
Public opinion and the end of appeasement in Britain and FranceDaniel Hucker · 2011- 📕Monumental space in the post-imperial novelRita Sakr · 2011
Art since 1900Hal Foster, Rosalind E. Krauss · 2004
Sexual ethicsAurel Kolnai · 2005
Archaeology of Atlantic Africa and the African DiasporaAkinwumi Ogundiran, Toyin Falola · 2006
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Self-supervised learning Self-supervised learning (SSL) is a paradigm in machine learning where a model is trained on a task using the data itself to generate supervisory signals…
- Representation learning In machine learning (ML), representation learning or feature learning is a set of techniques that allow a system to automatically discover the representations…
- Weight initialization In deep learning, weight initialization or parameter initialization describes the initial step in creating a neural network. A neural network contains…
- Geospatial foundation model A geospatial foundation model (GFM), also referred to as an Earth observation foundation model, represents a paradigm shift in Earth system modeling by…
- Outline of deep learning The following outline is provided as an overview of, and topical guide to, deep learning: Deep learning is a subfield of machine learning and artificial…
- MNIST database The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for…
- Vision transformer A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches (rather than text into…
- Graph neural network Graph neural networks (GNNs) are artificial neural networks designed for tasks whose inputs are graphs. Because graphs usually do not have a canonical…