Live briefing
Everything about Positional encoding
Deep learning technique
In deep learning, the positional encoding or position embedding is a component of sequence models, especially Transformers. Input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and the position of each token is encoded as a vector. Positional encoding is used because the standard attention mechanism used in Transformers by default treats all tokens symmetrically over all positions, thus effectively treating the input as a bag of tokens, rather than a sequence of tokens. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Positional encodingA 5-minute orientation
- Skim a recent paper: Improving position encoding of transformers for multivariate time series classificationInfluential work since 2022
- Pick up a classic book: Communications, Signal Processing, and SystemsThe most-reprinted book on the subject
- Read the landmark paper: Encoding and decoding of positional information in morphogen-dependent patterningCited 37 times
- Go sideways: BERT (language model)A closely related topic
- Go sideways: Transformer (deep learning)A closely related topic
📖 Definitions
No dictionary entry for this exact phrase.
🗂️ The subject at a glance
🏆 Landmark papers (most cited)
Encoding and decoding of positional information in morphogen-dependent patterning
Positional priming of pop-out: A relational-encoding account
Convolution-Embedded Vision Transformer With Elastic Positional Encoding for Pansharpening
Positional encoding in transformer-based time series models: a survey
Enhancing multivariate time-series anomaly detection with positional encoding mechanisms in transformers
Attention And Positional Encoding Are (Almost) All You Need For Shape Matching
Structural and positional ensembled encoding for Graph Transformer
Improved MLP Point Cloud Processing with High-Dimensional Positional Encoding
🆕 Influential since 2022
Improving position encoding of transformers for multivariate time series classification
Prediction of anticancer peptides based on an ensemble model of deep learning and machine learning using ordinal positional encoding
Safety and Outcomes of a Plasmid DNA Vaccine Encoding the ERBB2 Intracellular Domain in Patients With Advanced-Stage ERBB2-Positive Breast Cancer
HDTSLR: A Framework Based on Hierarchical Dynamic Positional Encoding for Sign Language Recognition
Positional trace encoding for next activity prediction in event logs
📚 Books worth reading
Communications, Signal Processing, and SystemsQilian Liang · 2012
Clean CodeRobert C. Martin · 2008
The web application hacker's handbookDafydd Stuttard, Marcus Pinto · 2007
Essentials of educational psychologyJeanne Ellis Ormrod · 2006
MetagenomicsWolfgang R. Streit, Rolf Daniel · 2010
Treaty Occupationay ToxicologySuelen Queiroz · 2010- 📕Adobe Flash CS4 ProfessionalRussell Chun · 2009
EthernetCharles E. Spurgeon, Joann Zimmerman · 2000
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- BERT (language model) Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google. It learns to represent…
- Transformer (deep learning) In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data…
- Vision-language model A vision–language model (VLM) is a type of artificial intelligence system that can jointly interpret and generate information from both images and text…
- Vision transformer A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches (rather than text into…
- Attention (machine learning) In machine learning, attention is a method that determines the importance of, or "attends to", each component in a sequence relative to the other components…
- Attention Is All You Need "Attention Is All You Need" is a 2017 research paper on machine learning authored by eight scientists and engineers working at Google. The paper introduced…
- Neural machine translation Neural machine translation (NMT) is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of…
- Training of large language models A large language model (LLM) needs to be trained. The training of an LLM has become more complex over time. Pretraining is the training stage where the…