Everything about Entropy (information theory)
Average uncertainty in variable's states
In information theory, the entropy of a random variable quantifies the average level of uncertainty or information associated with the variable's potential states or possible outcomes. This measures the expected amount of information needed to describe the state of the variable, considering the distribution of probabilities across all potential states. Given a discrete random variable, which may be any member within the set and is distributed according to, the entropy is where denotes the sum over the variable's possible values. The choice of base for, the logarithm, varies for different applications. Base 2 gives the unit of bits, while base e gives "natural units" nat, and base 10 gives units of "dits", "bans", or "hartleys". An equivalent definition of entropy is the expected value of the self-information of a variable. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Entropy (information theory)A 5-minute orientation
- Skim a recent paper: Shock compression and spallation damage of high-entropy alloy Al0.1CoCrFeNiInfluential work since 2022
- Pick up a classic book: Entropy Coders of the H.264/AVC StandardThe most-reprinted book on the subject
- Read the landmark paper: Entropy and data compression schemesCited 213 times
- Go sideways: Information contentA closely related topic
- Go sideways: Joint entropyA closely related topic
📖 Definitions
No dictionary entry for this exact phrase.
🏆 Landmark papers (most cited)
Entropy and data compression schemes
Shock compression response of high entropy alloys
Compression of Low Entropy Strings with Lempel--Ziv Algorithms
Acyclic edge-coloring using entropy compression
Modeling high-entropy transition metal alloys with alchemical compression
Relationship Between Entropy and Test Data Compression
Epsilon Entropy and Data Compression
Learning Context-Based Nonlocal Entropy Modeling for Image Compression
🆕 Influential since 2022
Shock compression and spallation damage of high-entropy alloy Al0.1CoCrFeNi
Phase transition in medium entropy alloy CoCrNi under quasi-isentropic compression
Shock compression and spallation of a medium-entropy alloy Fe40Mn20Cr20Ni20
Amorphization transformation in high-entropy alloy FeNiCrCoCu under shock compression
Multiscale Latent-Guided Entropy Model for LiDAR Point Cloud Compression
📚 Books worth reading
Entropy Coders of the H.264/AVC StandardXiaohua Tian · 2010
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Information content In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of…
- Joint entropy In information theory, joint entropy is a measure of the uncertainty associated with a set of variables. The joint Shannon entropy (in bits) of two discrete…
- Quantities of information The mathematical theory of information is based on probability theory and statistics, and measures information with several quantities of information,…
- Information theory Information theory is the mathematical study of the quantification, storage, and communication of a particular type of mathematically defined information…
- Typical set In information theory, the typical set is a set of sequences whose probability is approximately 2−nH(X), where H(X) is the entropy of the source and n…
- Rényi entropy In information theory, the Rényi entropy is a quantity that generalizes various notions of entropy, including Hartley entropy, Shannon entropy, collision…
- Conditional entropy In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable Y given that the…
- Differential entropy Differential entropy (also referred to as continuous entropy) in information theory is a property of absolutely continuous probability distributions which…