Virtual Lab: Introduction to Information Theory

Laboratory Objectives

Upon completion of this lab, students will be able to:

Quantify Information

Calculate the self-information and entropy of a discrete memoryless source based on symbol probabilities.

Source Coding

Understand the Shannon Source Coding Theorem and design efficient codes using Huffman coding principles.

Channel Capacity

Analyze the effect of noise on binary transmission and calculate the theoretical channel capacity limit.

Theoretical Background

1. Information Content & Entropy

Information theory, founded by Claude Shannon, quantifies information. The information content $I(x)$ of an event $x$ with probability $P(x)$ is defined as:

I(x) = -log₂(P(x)) Unit: Bits (if log base 2)

Entropy ($H$) is the average information content of a source. It represents the uncertainty or randomness.

H(X) = - Σ P(xᵢ) log₂(P(xᵢ))

If one symbol has $P=1$, entropy is 0 (no uncertainty).
For a binary source with equal probability ($P=0.5$), entropy is maximum at 1 bit/symbol.

2. Source Coding Theorem

The theorem states that the average code length $L$ must be greater than or equal to the entropy $H(X)$ for lossless compression.

H(X) ≤ L < H(X) + 1

Huffman coding is a greedy algorithm used to construct an optimal prefix code that approaches this entropy limit.

3. Channel Capacity

The Shannon-Hartley theorem defines the maximum rate at which information can be transmitted over a noisy channel without error.

C = B log₂(1 + SNR)

Where $C$ is capacity in bits/sec, $B$ is bandwidth, and $SNR$ is Signal-to-Noise Ratio. For a Binary Symmetric Channel (BSC) with error probability $p$:

C = 1 - H(p) Where H(p) is the binary entropy function of the error probability.

Experiment 1: Source Entropy & Coding

Visualize how probability distribution affects entropy and code efficiency.

Source Configuration

Adjust probabilities for symbols A, B, C, D. (Auto-normalized)

Calculated Entropy (H): 0.00 bits

Avg Code Length (L): 0.00 bits

Efficiency: 0%

Probability Distribution

Huffman Coding Tree

Tree visualization will appear here

Experiment 2: Binary Symmetric Channel (BSC)

Simulate data transmission over a noisy channel and observe the relationship between Error Probability ($p$) and Mutual Information.

Channel Transmission

Error Probability (p): 0.10

SOURCE (Input)

Noisy Channel

DESTINATION (Output)

Bits Sent

Errors

Error Rate

Channel Capacity

C = 1 - H(p)

Current Capacity

0.90 bits/use

Laboratory Procedure

Source Entropy Analysis

Navigate to Experiment 1.

Set the probability of Symbol A to 1.0 and others to 0. Observe the Entropy ($H=0$).
Adjust the source to a uniform distribution (A=0.25, B=0.25, C=0.25, D=0.25). Record the maximum Entropy.
Create a skewed distribution (e.g., A=0.7, B=0.2, C=0.1, D=0.0). Compare the Entropy to the uniform case.

Huffman Coding Efficiency

Observe the generated codes.

Note the Huffman codes generated for the skewed distribution. Which symbol gets the shortest code? Why?
Calculate the Average Code Length ($L$) manually and verify it against the simulation.
Calculate Coding Efficiency ($\eta = H/L$) and verify it is close to 100%.

Noisy Channel Simulation

Navigate to Experiment 2.

Set Error Probability ($p$) to 0.0. Send 20 bits. Verify 0% error.
Set $p$ to 0.5. Send bits. Observe the output is completely random relative to input. Check the Channel Capacity graph (should be 0).
Set $p$ to 0.1. Send 100 bits. Record the experimental error rate. Does it match $p$?

Report Generation

Compile your findings.

Include screenshots of the Entropy calculation for 3 different distributions.
Plot the theoretical Channel Capacity vs. Error Probability ($p$) curve.
Discuss the significance of $H=0$ and $C=0$ in physical communication systems.