Information Theory (信息论主题)

Study Note:
Theorem 2.7.4 (P31) The mutual information I(X;Y) is a convex function of p(y|x) for fixed p(x).
Hint: the key here is to consider I(X;Y) as function of p(x) and p(y|x) on one hand, and to use the relationship I(X;Y)=D(p(x,y)||p(x)p(y)) on the other hand.

To prove the convexity of I(X;Y)=I(p(x),p(y|x)) about p(y|x) is to demonstrate
I(p(x),up1(y|x)+(1-u)p2(y|x))<=uI(p(x),p1(y|x))+(1-u)I(p(x),p2(y|x)).

it is true because
I(p(x),up1(y|x)+(1-u)p2(y|x))
=D(u*p1(x,y)+(1-u)*p2(y|x)||p(x)(u*p1(y)+(1-u)*p2(y)))
<=uD(p1(x,y)||p(x)p1(y))+(1-u)D(p2(x,y)||p(x)p2(y))
=u*I(p(x),p1(y|x))+(1-u)I(p(x),p2(y|x))
[note that the expression of D(p||q) according to I(X;Y) is decided by the corresponding combine distribution of p(x,y) and marginals p(x) and p(y), so here basiclly is the change of form. Also note hat p1(x)=p2(x)=p(x)]