The Physics of Information: Thermodynamics, Shannon Entropy, and the Limits of Compression

Information is physical. Not in a hand-wavy, philosophical sense — in a hard, measurable, thermodynamic sense. In 1961, Rolf Landauer demonstrated that erasing a single bit of information must dissipate a minimum amount of heat into the surrounding environment, bounded below by:

$E_{min} = k_B T \ln 2$

This is Landauer’s principle. At room temperature (~300 K), that works out to roughly $2.85 \times 10^{-21}$ joules per bit — negligible by today’s standards, but a fundamental physical limit that modern processors are now approaching. The principle specifically applies to erasure, not writing: flipping a bit from 0 to 1 is reversible in principle, but erasing a bit (forcing it to a known state regardless of its previous value) destroys information and generates entropy as a consequence.

The practical takeaway for engineers: when a compression algorithm fails to shrink a random dataset, it isn’t a software bug. It’s the universe enforcing a constraint. To understand where that constraint comes from, you have to go back to 19th-century gas physics.

Boltzmann and the Counting of States

Classical thermodynamics, as Clausius and Carnot understood it, treated entropy as something you measured in steam engines — a bookkeeping variable for heat flows. Ludwig Boltzmann reframed the entire concept from the ground up. His insight was that macroscopic properties like temperature and pressure are statistical averages over an enormous number of underlying microscopic configurations.

The pressure your lungs feel when you breathe in isn’t some smooth, continuous property of air — it’s the aggregate of $10^{23}$ molecules bouncing around, each with its own position and momentum. Most of those microscopic arrangements produce the same macroscopic result. Boltzmann asked: how many arrangements are possible?

He encoded the answer in a single equation:

$S = k_B \ln(W)$

$S$ : thermodynamic entropy (J/K)
$k_B$ : Boltzmann constant, $1.380649 \times 10^{-23}$ J/K
$W$ : the number of distinct microstates consistent with the observed macrostate

Consider a perfect crystal at absolute zero. Every atom is locked in place. There is exactly one valid arrangement: $W = 1$ , so $S = k_B \ln(1) = 0$ . No uncertainty about the system’s state, no entropy. Now heat that crystal until it vaporizes. The atoms roam freely. $W$ becomes something like $10^{10^{23}}$ — unimaginably large — and entropy skyrockets accordingly.

From an information perspective, entropy measures exactly how much you don’t know. High $W$ means the system could be in any of a huge number of states. Low $W$ means the system’s configuration is nearly determined. This is not an analogy — it is literally the same mathematics Shannon would rediscover sixty years later.

From Gas Molecules to Binary Streams: Deriving Shannon Entropy

In 1948, Claude Shannon was working on a problem at Bell Labs that had nothing to do with thermodynamics on the surface: how much information can a noisy telegraph channel carry? He needed a rigorous, quantitative definition of “information” — one that didn’t depend on semantics, only on the statistics of the source.

He arrived at the same formula as Boltzmann.

The derivation is straightforward. Start with a physical system that has $W$ equally probable microstates. The probability of landing in any particular microstate is $p = 1/W$ , so $W = 1/p$ . Substitute into Boltzmann:

$S = k_B \ln\left(\frac{1}{p}\right) = -k_B \ln(p)$

This gives the entropy contribution of a single, equally probable state. But real information sources — text, binary executables, network traffic — don’t have uniformly distributed symbols. The letter ‘e’ appears far more often than ‘z’ in English text. To handle this, we take the expected value of entropy across all possible symbols, each with its own probability $p_i$ :

$S = -k_B \sum_{i} p_i \ln(p_i)$

This is the ensemble average, the thermodynamic entropy of the full distribution.

Shannon then made two adaptations to convert this into a tool for communication engineering:

Dropped $k_B$ : The Boltzmann constant ties entropy to physical units (J/K). Shannon wanted a unit-free measure of information, so he discarded it.
Changed $\ln$ to $\log_2$ : Switching the logarithm base from $e$ to $2$ re-expresses entropy in bits rather than nats. Since digital systems use binary, this is the natural unit.

The result is Shannon entropy:

$H(X) = -\sum_{i=1}^{n} p(x_i) \log_2 p(x_i)$

where $H(X)$ is measured in bits per symbol. The formula is identical in structure to Boltzmann’s — the physics and the information theory are not analogous, they are the same mathematical object, viewed from different domains.

Shannon’s Source Coding Theorem: The Hard Compression Limit

Shannon didn’t just define entropy — he proved what it means for compression. His Source Coding Theorem states:

Any lossless compression scheme must, on average, use at least $H(X)$ bits per symbol to represent the source $X$ .

This is a mathematical impossibility result. You cannot compress below $H(X)$ without losing information. If a file has $N$ symbols and per-symbol entropy $H$ , the theoretical minimum size is exactly $N \times H$ bits.

Let’s work through a concrete example. A source emits four symbols with the following probabilities:

Symbol	Probability
A	0.5
B	0.25
C	0.125
D	0.125

Computing $H$ :

$H = -\left(0.5 \log_2 0.5 + 0.25 \log_2 0.25 + 0.125 \log_2 0.125 + 0.125 \log_2 0.125\right)$ $H = -\left(0.5(-1) + 0.25(-2) + 0.125(-3) + 0.125(-3)\right)$ $H = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 \text{ bits/symbol}$

A flat 2-bit encoding (00/01/10/11) wastes 0.25 bits per symbol. ASCII would waste 6.25. Huffman coding — which assigns shorter codes to more probable symbols — achieves close to this limit:

Symbol	Huffman Code	Length
A	`0`	1 bit
B	`10`	2 bits
C	`110`	3 bits
D	`111`	3 bits

Average length: $0.5(1) + 0.25(2) + 0.125(3) + 0.125(3) = 1.75$ bits/symbol — exactly hitting the Shannon bound in this particular case.

Huffman isn’t always this lucky. When probabilities aren’t negative powers of two, Huffman produces integer-length codes that can’t precisely hit fractional entropy values. For a source with $H = 2.3$ bits/symbol, Huffman might achieve 2.5 and no further.

Arithmetic coding solves this by encoding an entire message as a single rational number in the interval $[0, 1)$ . Each symbol narrows the interval proportionally to its probability, allowing the encoder to accumulate fractional bits continuously rather than rounding each symbol to an integer code length. The result approaches the Shannon limit arbitrarily closely as the message length grows. The trade-off is computational cost and the need to process sequences rather than individual symbols — which is why modern compressors (zstd, brotli) combine an arithmetic/ANS coder with a context model that feeds it accurate probability estimates.

The Channel Capacity Connection: Where Bandwidth Comes From

The title of this article promises network bandwidth, so let’s honor that. Shannon also proved the Channel Capacity Theorem (the Shannon-Hartley theorem), which extends the entropy framework to noisy transmission channels:

$C = B \log_2\left(1 + \frac{S}{N}\right)$

$C$ : channel capacity in bits per second
$B$ : bandwidth of the channel in Hz
$S/N$ : signal-to-noise ratio (linear scale, not dB)

This is the absolute upper bound on how much information can be transmitted over a physical channel, regardless of the encoding scheme. A fiber link with 10 GHz of bandwidth and a 30 dB SNR has a theoretical capacity of roughly $10^{10} \times \log_2(1001) \approx 100$ Gbps. No protocol, no modulation scheme, no hardware trick can exceed this.

The relationship to compression entropy is direct. Shannon entropy tells you the minimum bits needed to represent a source. Channel capacity tells you the maximum bits a channel can carry per second. The two together define the fundamental throughput you can achieve: compress your source to its entropy limit, then transmit at the channel’s capacity limit. Everything beyond that — TCP windowing, FEC, QoS — is engineering to approach those physics-defined bounds, not to exceed them.

In practice, Wi-Fi 6E pushing 9.6 Gbps across 160 MHz channels is operating at a fraction of the Shannon limit because real-world SNR, interference, and protocol overhead eat into the theoretical maximum. The formula explains why more spectrum (larger $B$ ) or better antennas (higher $S/N$ ) improve throughput — and exactly by how much.

Practical Application: Entropy as a Binary Analysis Tool

In malware analysis and reverse engineering, entropy measurement is a first-pass heuristic that tells you a lot about a binary before you ever open a disassembler. The logic follows directly from Shannon: strong encryption produces output that’s indistinguishable from a uniform random distribution over all 256 byte values. A section with entropy near 8.0 bits/byte is almost certainly encrypted or compressed. A section at 6.0–6.5 is typical compiled code — structured, but not maximally random. A section below 4.0 is highly structured data (lookup tables, strings, resource blobs).

The following implementation calculates Shannon entropy over arbitrary binary files. It uses mmap rather than read() to avoid unnecessary kernel-to-userspace copies, and madvise(MADV_SEQUENTIAL) to hint the kernel prefetcher for a single-pass scan — meaningful on large files:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <math.h>

double calculate_entropy(const unsigned char *data, size_t length) {
    if (length == 0) return 0.0;

    size_t byte_counts[256] = {0};

    for (size_t i = 0; i < length; i++) {
        byte_counts[data[i]]++;
    }

    double entropy = 0.0;
    for (int i = 0; i < 256; i++) {
        if (byte_counts[i] > 0) {
            double p = (double)byte_counts[i] / (double)length;
            entropy -= p * log2(p);
        }
    }
    return entropy;
}

int main(int argc, char *argv[]) {
    if (argc != 2) {
        fprintf(stderr, "Usage: %s <file>\n", argv[0]);
        return 1;
    }

    int fd = open(argv[1], O_RDONLY);
    if (fd < 0) {
        perror("open");
        return 1;
    }

    struct stat st;
    if (fstat(fd, &st) < 0) {
        perror("fstat");
        close(fd);
        return 1;
    }

    if (st.st_size == 0) {
        printf("Entropy: 0.000000 bits/byte\n");
        close(fd);
        return 0;
    }

    unsigned char *data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
    if (data == MAP_FAILED) {
        perror("mmap");
        close(fd);
        return 1;
    }

    // Hint the kernel to prefetch sequentially; reduces page fault stalls
    // on large files significantly.
    madvise(data, st.st_size, MADV_SEQUENTIAL);

    double h = calculate_entropy(data, st.st_size);
    printf("Shannon Entropy: %.6f bits/byte\n", h);

    munmap(data, st.st_size);
    close(fd);
    return 0;
}

Compile with:

gcc -O2 -o entropy entropy.c -lm

Run it against a few reference targets to build intuition:

./entropy /dev/urandom          # → ~8.0  (theoretical max, random)
./entropy /bin/ls               # → ~6.1  (compiled ELF, typical)
./entropy archive.tar.gz        # → ~7.9  (already compressed)
./entropy config.json           # → ~4.5  (structured text, lots of repetition)

A .tar.gz file reading close to 8.0 should not be re-compressed — you’ll add CPU cycles for essentially zero gain, because the source is already at (or past) the entropy limit for the byte distribution. This is something naive compression pipelines get wrong.

What This Means in Practice

The core point is that entropy isn’t a metric you optimize toward — it’s a wall you cannot pass. Huffman, LZ77, zstd, brotli, LZMA: all of these are different strategies for approaching the Shannon bound, not for breaking it. When a compression algorithm “can’t help,” it’s because the source entropy is already close to 8 bits/byte for that data. When a channel “can’t go faster,” it’s because you’ve hit the Shannon-Hartley capacity for that SNR and bandwidth combination.

Landauer’s principle closes the loop back to physics: the information you can’t compress, you eventually have to erase, and erasure has a thermodynamic cost. As transistor counts climb toward physical limits and operating temperatures stop scaling down, Landauer’s bound becomes not just theoretical but an engineering constraint. Modern near-threshold computing research is already trying to run gates at kT-scale energies specifically to approach that limit from below.

The mathematics connecting Boltzmann’s gas molecules to your network throughput and your malware scanner is not a metaphor. It is one continuous derivation — the same equation, applied at different scales.