Foundations of Information Theory
What is information, and how do we measure it? This is a growing collection of material that builds information theory from first principles, starting from one question: how much does a receiver actually learn from a noisy signal? It is written for engineering students and engineers who want to understand the ideas behind the formulas. Basic probability is the only prerequisite.
Read online
The ideas of the chapter, in four short parts. Each part builds on the one before it.
- What is information? A noisy channel and a puzzle Information as a reduction in uncertainty, two small channels, and a puzzle about how much gets through.
- Why does the entropy formula have a logarithm? Surprise, the logarithm, and entropy as the average surprise of a source.
- What is conditional entropy? Solving the noisy-channel puzzle Bayes’ rule by counting, the uncertainty that remains after an observation, and the answer to the puzzle.
- Mutual information explained Why the information shared by sender and receiver is the same from both sides, and a first look at capacity.
The Chapter
The full chapter, with every derivation and a set of worked problems. It is a working draft and is updated regularly.
Lecture Slides
Slides for a 90-minute lecture based on the chapter: a reading version, and a presentation version that reveals each step click by click.
Slides: reading version (PDF) Slides: presentation version (PDF)
Video Lecture
Coming soon.
© 2026 Balaji Dhayabaran. The chapter, slides and these pages are licensed under CC BY-NC-ND 4.0: you may share them for non-commercial purposes, with credit, without modification. Teachers and students are welcome to use them in class. Follow Kanithakalam on LinkedIn for new parts as they appear.