More on Mutual Information

More on Mutual Information (MI)

Recall that, in communication system information source emits X into information channel which in turn emits Y.

observing output Y in communication system

We therefore observe Y, i.e., we know Y. The question then is:

What does knowing Y tell us about X?

This Y↔X relationship is called mutual information.

Which is,

"reduction in uncertainty of X given the knowledge of Y".

There are at least two approaches to deriving the mutual information:

Shannon's approach (using Entropy Algebra) I(X; Y) = H(X) − H(X|Y)
Statistical approach (using Probability distribution).

$I(X; Y) = for all x sub i, for all y sub j, product of p(x sub i, y sub j) and log base 2 of (p(x sub i, y sub j) over (product of p(x sub i) and p(y sub j)))$

where, p(x_i) p(y_j) = 0 because if X & Y were statistically independent I(X;Y) = 0.

Shannon's approach was illustrated in lecture 3.

❶

Statistical approach of deriving I(X; Y) and also showing I(X; Y) = I(Y; X).

Since, mutual information is the relative entropy between the joint distribution and the product distribution p(x_i) p(y_j),

$I(X; Y) = for all x sub i, for all y sub j, product of p(x sub i, y sub j) and log base 2 of (p(x sub i, y sub j) over (product of p(x sub i) and p(y sub j)))$

But, p(x_i, y_j) = p(x_i|y_j) p(y_j) = p(y_j|x_i) p(x_i) And it is easier to find conditional probability than joint probability and hence for more convenient calculation we use the form p(x_i|y_j) p(y_j) or p(y_j|x_i) p(x_i).

Thus,

$I(X; Y) = for all x sub i, for all y sub j, sum of products of p(x sub i given y sub j) times p(y sub j) times log base 2 of (p(x sub i given y sub j) over p(x sub i) OR I(X; Y) = for all x sub i, for all y sub j, sum of products of p(y sub j given x sub i) times p(x sub i) times log base 2 of (p(y sub j given x sub i) over p(y sub j)$