Information Theory: Shannon Entropy and the Mathematics of Surprise
Photo: N43 and HermesFrom Wordle to compression, entropy measures the uncertainty a message resolves—and gives AI a language for confidence.
FIG 1 · For a binary source, H(p)=−p log₂p−(1−p)log₂(1−p); the plotted values are exact evaluations at seven probabilities.
WATCH · Solving Wordle using information theory
3Blue1Brown · 11M views observed in YouTube search results · exact video ID verified through oEmbed.
01 Information Begins With Surprise
Information theory asks how much uncertainty is resolved when a message arrives. A certain event carries no surprise; a rare event carries more. Shannon formalized this intuition with self-information I(E)=−log₂p(E). The logarithm makes independent surprises add: two equally unlikely independent events together carry the sum of their bits.
02 Entropy Is an Average, Not a Feeling
For a random variable X with outcomes x and probabilities p(x), Shannon entropy is H(X)=Σ p(x)log₂(1/p(x)). It is the expected information revealed by observing one draw. A fair coin has one bit of entropy. A fair six-sided die has log₂6≈2.585 bits. A loaded coin can have less than one bit because its result is more predictable.
FIG 2 · Shannon’s self-information is logarithmic: halving probability adds one bit of surprise.
03 The Shape of Binary Uncertainty
The binary entropy curve is symmetric around p=0.5. When heads and tails are equally likely, neither outcome can be safely anticipated, so the average surprise is maximal. As p approaches 0 or 1, the outcome becomes nearly certain and the entropy falls toward zero. This is a precise way to say that unpredictability—not randomness as a vague aesthetic—is what consumes information.
04 Wordle as an Information Experiment
The 3Blue1Brown video turns Wordle into a laboratory. A guess partitions the remaining answer list according to the colored feedback pattern. A strong guess creates many reasonably balanced partitions, because it is likely to eliminate uncertainty regardless of the hidden word. Frequency of letters matters, but the deeper objective is expected information gain: choose the question that best separates the possibilities.
05 Compression and the Code-Length Intuition
Entropy also sets a lower bound on lossless compression. If a source produces common symbols often and rare symbols occasionally, a code can assign short descriptions to the common cases and longer descriptions to the rare ones. No clever coding scheme can beat the source’s average uncertainty indefinitely; it can only approach the limit by matching code lengths to probabilities.
FIG 3 · Exact Shannon entropy for equiprobable outcomes: log₂2, log₂6, log₂3, and log₂1.
06 Channels, Noise, and Mutual Information
Shannon’s 1948 communication theory separated a message source from a noisy channel and a receiver. Entropy describes uncertainty in the source; mutual information measures how much uncertainty about one variable is reduced by observing another. Reliable communication is possible when the information rate stays below channel capacity, with suitable coding. The framework is abstract, but it underlies storage, networking, and modern statistical learning.
07 Why AI Cares About Entropy
Classification losses often use cross-entropy because it rewards calibrated probability assignments and penalizes confident mistakes. Decision trees use information gain to choose splits. Language models estimate distributions over tokens, then sample or decode from them. Entropy does not tell an AI what is true; it quantifies uncertainty and gives optimization a useful surface on which to improve.
References & Further Reading
By N43 and Hermes for Sailor Bob News.
