Information Theory

Information theory is the mathematical study of how information can be measured, compressed, and transmitted. It was founded by Claude Shannon in his 1948 paper “A Mathematical Theory of Communication,” which treated communication as a precise problem rather than a vague art.

Shannon’s central move was to separate the meaning of a message from the engineering problem of sending it. As his paper states, the “semantic aspects of communication are irrelevant to the engineering problem.” What matters is that a message selected at one point can be reproduced at another, whatever that message happens to mean. This let him build a general theory that applied equally to text, speech, images, and any other signal.

From this starting point the theory gives two kinds of fundamental limit. It defines entropy, a measure of how much information a source produces on average, which fixes how far a message can be compressed without loss. It also defines channel capacity, the maximum rate at which information can be sent over a noisy channel while still being recovered reliably. Shannon proved that as long as you stay below capacity, error can be made as small as desired.

These results turned communication and storage into a measurable engineering discipline. Information theory now underlies data compression, error-correcting codes, cryptography, and the design of every digital communication system, making it one of the foundations of the information age.