1. The document contains 10 problems related to information theory concepts like entropy, mutual information, and channel models.
2. Problem 1 involves calculating the entropy of a source with 6 states and given probabilities. Problem 2 asks to find the information of 3 dice rolling cases.
3. Problem 3 involves calculating the entropy of a fair coin flip experiment until the first head.
1. The document contains 10 problems related to information theory concepts like entropy, mutual information, and channel models.
2. Problem 1 involves calculating the entropy of a source with 6 states and given probabilities. Problem 2 asks to find the information of 3 dice rolling cases.
3. Problem 3 involves calculating the entropy of a fair coin flip experiment until the first head.
1. The document contains 10 problems related to information theory concepts like entropy, mutual information, and channel models.
2. Problem 1 involves calculating the entropy of a source with 6 states and given probabilities. Problem 2 asks to find the information of 3 dice rolling cases.
3. Problem 3 involves calculating the entropy of a fair coin flip experiment until the first head.
1. The document contains 10 problems related to information theory concepts like entropy, mutual information, and channel models.
2. Problem 1 involves calculating the entropy of a source with 6 states and given probabilities. Problem 2 asks to find the information of 3 dice rolling cases.
3. Problem 3 involves calculating the entropy of a fair coin flip experiment until the first head.
1. A source has 6 independent states, the probabilities are PA = 0.5 PB = 0.
25
PC = 0.125 PD = PE = 0.05 PF = 0.025 , respectively.
a) To calculate H ( X ) .
b) How much information the message ABABBA and FDDFDF contains?
2. Throw two dices, when the sum is 2, 7, and 11. Please find the information of 3 cases. 3. Coin flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. Find the entropy H(X) in bits. The following expressions may be useful: ¥ ¥ r r å rn = n =1 1- r , å nr n =1 n = (1 - r ) 2 4. Entropy of functions. Let X be a random variable taking on a finite number of values. What is the (general) inequality relationship of H(X) and H(Y) if (a) Y = 2X ? (b) Y = cos X ? 5. Please draw the model of ternary symmetric channel. (with 3 inputs and 3 outputs) when (1) completely noise channel (2) noise-less channel 6. Please find I ( x = 0 ; y = 1 ) and I ( X ;Y ) of the followed binary channel
X Y P(0) = 0.2 0 0.7 0 0.3
0.1 P(1) = 0.8 1 1 0.9 7. Example of joint entropy. Let p(x,y) be given by X Y 0 1 0 1/3 1/3 1 0 1/3 Find (a) H(X), H(Y). (b) H(X|Y), H(Y|X). (c) H(X,Y). (d) H(Y) - H(Y|X). (e) I(X;Y). 8. In a female population X, consisting of 1/4 blondes, 1/2 brunettes, and 1/4 redheads, blondes are always on time for engagements, redheads are always late, and each brunette always flips an unbiased coin for each engagement to decide whether to be prompt or tardy. （a） How much information is given by the statement “x, a member of X, arrived on time” about each of the following propositions: (1) x is a blonde, (2) x is a brunette, (3) x is a redhead. （b） How much information is given by the statement “x, a member of X, arrived on time for three engagements in a row” about the proposition “x is a brunette”? 9. A source X produces letters from a three-symbol alphabet with the probability assignment PX(0)=1/4, PX(1)=1/4, PX(2)=1/2. Each source letter x is directly transmitted through two channels simultaneously with outputs y and z and the transition probabilities indicated below: x P(z|x) z x P(y|x) y 0 1 0 0 1 0 1/2 1 1 1 1 1 1 1/2 1 2 2
(Note that this could be considered as a single channel with output yz). Calculate H(X), H(Y), H(Z), H(YZ), I(X;Y), I(X;Z), I(X;YZ). Interpret the mutual information expressions. 10. Please find the I(X; Y) of the followed channel P( x1 ) = p 1-a x1 y1 a P(x2 ) = 1 - p b x2 y2 1- b
