Since
Why is ?
Thus p(xi) = p(xi) only if .
❷
H(C) =
H(X) +
H(Y|X)
is derived from the fact that
p(xi, yj) = p(yj|xi) p(xi)
But, p(xi, yj) is also
p(xi, yj) = p(xi|yj) p(yj)
which will therefore lead us to
H(C) =
H(X,Y)
= H(Y) +
H(X|Y)
Thus,
H(X) +
H(Y|X)
= H(Y) +
H(X|Y)
This is the heart of entropy algebra.
❸
☛
Alternatively,
Heart of Entropy Algebra
Recall, that the chain rule
Example application of the chain rule.
cij =
〈xi, yj, zk〉
Then,
H(C) =
H(X, Y, Z)
= H(X, 〈Y, Z〉)
applying chain rule
= H(X) +
H(〈Y, Z〉|X)
= H(X) +
H(Y, Z|X)
applying chain rule
= H(X) +
H(Y|X) +
H(Z|X, Y)
Generalization of Chain Rule
Thus resulting in
H(A0, A1, …,
An−1)
= H(A0) +
H(A1|A0) + H(A2|A0, A1) + …
H(An−1|A0, A1, A2, … An−2)
❹
Let,
Next:
Mutual Information (p:4) ➽