Live briefing
Everything about Multimodal learning
Machine learning methods using multiple input modalities
Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Multimodal learningA 5-minute orientation
- Skim a recent paper: Multimodal Learning With Transformers: A SurveyInfluential work since 2022
- Pick up a classic book: Technology, literacy, learningThe most-reprinted book on the subject
- Read the landmark paper: Dictionary learning for integrative, multimodal and scalable single-cell analysisCited 6,482 times
- Go sideways: Vision-language modelA closely related topic
- Go sideways: Contrastive language–image pre-trainingA closely related topic
📖 Definitions
No dictionary entry for this exact phrase.
🗂️ The subject at a glance
🏆 Landmark papers (most cited)
Dictionary learning for integrative, multimodal and scalable single-cell analysis
Interactive Multimodal Learning Environments
Multimodal Emotion Recognition using Deep Learning
Multimodal data capabilities for learning: What can multimodal data tell us about learning?
Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
Multimodal learning with graphs
TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis
Multimodal learning analytics for game‐based learning
🆕 Influential since 2022
📚 Books worth reading
Technology, literacy, learningCarey Jewitt · 2005
Multimodal teaching and learningGunther R. Kress, Carey Jewitt · 2001
Multimodality, Learning and CommunicationJeff Bezemer, Gunther Kress · 2015- 📕Multimodal Composing in ClassroomsSuzanne M. Miller, Mary B. McVee · 2013
Multimodality and Social SemiosisMargit Böck, Norbert Pachler · 2013- 📕Designing Learning for Multimodal LiteracyFei Victor Lim, Lydia Tan-Chia · 2022
- 📕Multimodal Literacies Across Digital Learning ContextsMaria Grazia Sindoni, Ilaria Moschini · 2021
- 📕Multimodal Signs of LearningShirley Palframan · 2021
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Vision-language model A vision–language model (VLM) is a type of artificial intelligence system that can jointly interpret and generate information from both images and text…
- Contrastive language–image pre-training Contrastive language–image pre-training (CLIP) is a technique for training a pair of neural network models, one for image understanding and one for text…
- Transformer (deep learning) In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data…
- Vision transformer A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches (rather than text into…
- Geospatial foundation model A geospatial foundation model (GFM), also referred to as an Earth observation foundation model, represents a paradigm shift in Earth system modeling by…
- Attention Is All You Need "Attention Is All You Need" is a 2017 research paper on machine learning authored by eight scientists and engineers working at Google. The paper introduced…
- Stable Diffusion Stable Diffusion is a deep learning, text-to-image model released in 2022 based on diffusion techniques. The generative AI technology is the premier product…
- Self-supervised learning Self-supervised learning (SSL) is a paradigm in machine learning where a model is trained on a task using the data itself to generate supervisory signals…