# 条件互信息

In probability theory, particularly information theory, the conditional mutual information[1][2] is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.

## Definition 定义

For random variables $\displaystyle{ X }$, $\displaystyle{ Y }$, and $\displaystyle{ Z }$ with support sets $\displaystyle{ \mathcal{X} }$, $\displaystyle{ \mathcal{Y} }$ and $\displaystyle{ \mathcal{Z} }$, we define the conditional mutual information as

$\displaystyle{ I(X;Y|Z) = \int_\mathcal{Z} D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} ) dP_{Z} }$

This may be written in terms of the expectation operator:

$\displaystyle{ I(X;Y|Z) = \mathbb{E}_Z [D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} )] }$.

Thus $\displaystyle{ I(X;Y|Z) }$ is the expected (with respect to $\displaystyle{ Z }$) Kullback–Leibler divergence from the conditional joint distribution $\displaystyle{ P_{(X,Y)|Z} }$ to the product of the conditional marginals $\displaystyle{ P_{X|Z} }$ and $\displaystyle{ P_{Y|Z} }$. Compare with the definition of mutual information.

## In terms of pmf's for discrete distributions 关于离散分布的概率质量函数

For discrete random variables $\displaystyle{ X }$, $\displaystyle{ Y }$, and $\displaystyle{ Z }$ with support sets $\displaystyle{ \mathcal{X} }$, $\displaystyle{ \mathcal{Y} }$ and $\displaystyle{ \mathcal{Z} }$, the conditional mutual information $\displaystyle{ I(X;Y|Z) }$ is as follows

$\displaystyle{ I(X;Y|Z) = \sum_{z\in \mathcal{Z}} p_Z(z) \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} p_{X,Y|Z}(x,y|z) \log \frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)} }$

where the marginal, joint, and/or conditional probability mass functions are denoted by $\displaystyle{ p }$ with the appropriate subscript. This can be simplified as

$\displaystyle{ I(X;Y|Z) = \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} p_{X,Y,Z}(x,y,z) \log \frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}. }$

## In terms of pdf's for continuous distributions 关于连续分布的概率密度函数

For (absolutely) continuous random variables $\displaystyle{ X }$, $\displaystyle{ Y }$, and $\displaystyle{ Z }$ with support sets $\displaystyle{ \mathcal{X} }$, $\displaystyle{ \mathcal{Y} }$ and $\displaystyle{ \mathcal{Z} }$, the conditional mutual information $\displaystyle{ I(X;Y|Z) }$ is as follows

$\displaystyle{ I(X;Y|Z) = \int_{\mathcal{Z}} \bigg( \int_{\mathcal{Y}} \int_{\mathcal{X}} \log \left(\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)}\right) p_{X,Y|Z}(x,y|z) dx dy \bigg) p_Z(z) dz }$

where the marginal, joint, and/or conditional probability density functions are denoted by $\displaystyle{ p }$ with the appropriate subscript. This can be simplified as

$\displaystyle{ I(X;Y|Z) = \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}} \log \left(\frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}\right) p_{X,Y,Z}(x,y,z) dx dy dz. }$

## Some identities 部分特性

Alternatively, we may write in terms of joint and conditional entropies as[3]

$\displaystyle{ I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z) = H(X|Z) - H(X|Y,Z) = H(X|Z)+H(Y|Z)-H(X,Y|Z). }$

This can be rewritten to show its relationship to mutual information

$\displaystyle{ I(X;Y|Z) = I(X;Y,Z) - I(X;Z) }$

usually rearranged as the chain rule for mutual information

$\displaystyle{ I(X;Y,Z) = I(X;Z) + I(X;Y|Z) }$

Another equivalent form of the above is[5]

$\displaystyle{ I(X;Y|Z) = H(Z|X) + H(X) + H(Z|Y) + H(Y) - H(Z|X,Y) - H(X,Y) - H(Z) = I(X;Y) + H(Z|X) + H(Z|Y) - H(Z|X,Y) - H(Z) }$

Like mutual information, conditional mutual information can be expressed as a Kullback–Leibler divergence:

$\displaystyle{ I(X;Y|Z) = D_{\mathrm{KL}}[ p(X,Y,Z) \| p(X|Z)p(Y|Z)p(Z) ]. }$

Or as an expected value of simpler Kullback–Leibler divergences:

$\displaystyle{ I(X;Y|Z) = \sum_{z \in \mathcal{Z}} p( Z=z ) D_{\mathrm{KL}}[ p(X,Y|z) \| p(X|z)p(Y|z) ] }$,
$\displaystyle{ I(X;Y|Z) = \sum_{y \in \mathcal{Y}} p( Y=y ) D_{\mathrm{KL}}[ p(X,Z|y) \| p(X|Z)p(Z|y) ] }$.

## More general definition 其他通用定义

A more general definition of conditional mutual information, applicable to random variables with continuous or other arbitrary distributions, will depend on the concept of regular conditional probability. (See also. [7][8])

Let $\displaystyle{ (\Omega, \mathcal F, \mathfrak P) }$ be a probability space, and let the random variables $\displaystyle{ X }$, $\displaystyle{ Y }$, and $\displaystyle{ Z }$ each be defined as a Borel-measurable function from $\displaystyle{ \Omega }$ to some state space endowed with a topological structure.

$\displaystyle{ (\Omega, \mathcal F, \mathfrak P) }$为一个 概率空间 Probability space ，并将随机变量$\displaystyle{ X }$, $\displaystyle{ Y }$$\displaystyle{ Z }$分别定义为一个从$\displaystyle{ \Omega }$到具有拓扑结构的状态空间的 波莱尔可测函数 Borel-measurable function

Consider the Borel measure (on the σ-algebra generated by the open sets) in the state space of each random variable defined by assigning each Borel set the $\displaystyle{ \mathfrak P }$-measure of its preimage in $\displaystyle{ \mathcal F }$. This is called the pushforward measure $\displaystyle{ X _* \mathfrak P = \mathfrak P\big(X^{-1}(\cdot)\big). }$ The support of a random variable is defined to be the topological support of this measure, i.e. $\displaystyle{ \mathrm{supp}\,X = \mathrm{supp}\,X _* \mathfrak P. }$

Now we can formally define the conditional probability measure given the value of one (or, via the product topology, more) of the random variables. Let $\displaystyle{ M }$ be a measurable subset of $\displaystyle{ \Omega, }$ (i.e. $\displaystyle{ M \in \mathcal F, }$) and let $\displaystyle{ x \in \mathrm{supp}\,X. }$ Then, using the disintegration theorem:

$\displaystyle{ \mathfrak P(M | X=x) = \lim_{U \ni x} \frac {\mathfrak P(M \cap \{X \in U\})} {\mathfrak P(\{X \in U\})} \qquad \textrm{and} \qquad \mathfrak P(M|X) = \int_M d\mathfrak P\big(\omega|X=X(\omega)\big), }$

where the limit is taken over the open neighborhoods $\displaystyle{ U }$ of $\displaystyle{ x }$, as they are allowed to become arbitrarily smaller with respect to set inclusion.

$\displaystyle{ x }$的开放邻域$\displaystyle{ U }$处取极限，因为相对于 集包含 Set inclusion，它们可以任意变小。

Finally we can define the conditional mutual information via Lebesgue integration:

$\displaystyle{ I(X;Y|Z) = \int_\Omega \log \Bigl( \frac {d \mathfrak P(\omega|X,Z)\, d\mathfrak P(\omega|Y,Z)} {d \mathfrak P(\omega|Z)\, d\mathfrak P(\omega|X,Y,Z)} \Bigr) d \mathfrak P(\omega), }$

where the integrand is the logarithm of a Radon–Nikodym derivative involving some of the conditional probability measures we have just defined.

## Note on notation 注释符号

In an expression such as $\displaystyle{ I(A;B|C), }$ $\displaystyle{ A, }$ $\displaystyle{ B, }$ and $\displaystyle{ C }$ need not necessarily be restricted to representing individual random variables, but could also represent the joint distribution of any collection of random variables defined on the same probability space. As is common in probability theory, we may use the comma to denote such a joint distribution, e.g. $\displaystyle{ I(A_0,A_1;B_1,B_2,B_3|C_0,C_1). }$ Hence the use of the semicolon (or occasionally a colon or even a wedge $\displaystyle{ \wedge }$) to separate the principal arguments of the mutual information symbol. (No such distinction is necessary in the symbol for joint entropy, since the joint entropy of any number of random variables is the same as the entropy of their joint distribution.)

## Properties 属性

### Nonnegativity 非负性

It is always true that

$\displaystyle{ I(X;Y|Z) \ge 0 }$,

for discrete, jointly distributed random variables $\displaystyle{ X }$, $\displaystyle{ Y }$ and $\displaystyle{ Z }$. This result has been used as a basic building block for proving other inequalities in information theory, in particular, those known as Shannon-type inequalities. Conditional mutual information is also non-negative for continuous random variables under certain regularity conditions.[11]

$\displaystyle{ I(X;Y|Z) \ge 0 }$,

### Interaction information 交互信息

Conditioning on a third random variable may either increase or decrease the mutual information: that is, the difference $\displaystyle{ I(X;Y) - I(X;Y|Z) }$, called the interaction information, may be positive, negative, or zero. This is the case even when random variables are pairwise independent. Such is the case when:

$\displaystyle{ X \sim \mathrm{Bernoulli}(0.5), Z \sim \mathrm{Bernoulli}(0.5), \quad Y=\left\{\begin{array}{ll} X & \text{if }Z=0\\ 1-X & \text{if }Z=1 \end{array}\right. }$

in which case $\displaystyle{ X }$, $\displaystyle{ Y }$ and $\displaystyle{ Z }$ are pairwise independent and in particular $\displaystyle{ I(X;Y)=0 }$, but $\displaystyle{ I(X;Y|Z)=1. }$

$\displaystyle{ X \sim \mathrm{Bernoulli}(0.5), Z \sim \mathrm{Bernoulli}(0.5), \quad Y=\left\{\begin{array}{ll} X & \text{if }Z=0\\ 1-X & \text{if }Z=1 \end{array}\right. }$

$\displaystyle{ X }$, $\displaystyle{ Y }$$\displaystyle{ Z }$是成对独立的，特别是$\displaystyle{ I(X;Y)=0 }$，不过这里$\displaystyle{ I(X;Y|Z)=1. }$

### Chain rule for mutual information 互信息的链式法则

$\displaystyle{ I(X;Y,Z) = I(X;Z) + I(X;Y|Z) }$

## Multivariate mutual information 多元互信息

The conditional mutual information can be used to inductively define a multivariate mutual information in a set- or measure-theoretic sense in the context of information diagrams. In this sense we define the multivariate mutual information as follows:

$\displaystyle{ I(X_1;\ldots;X_{n+1}) = I(X_1;\ldots;X_n) - I(X_1;\ldots;X_n|X_{n+1}), }$

Where 其中

$\displaystyle{ I(X_1;\ldots;X_n|X_{n+1}) = \mathbb{E}_{X_{n+1}} [D_{\mathrm{KL}}( P_{(X_1,\ldots,X_n)|X_{n+1}} \| P_{X_1|X_{n+1}} \otimes\cdots\otimes P_{X_n|X_{n+1}} )]. }$

This definition is identical to that of interaction information except for a change in sign in the case of an odd number of random variables. A complication is that this multivariate mutual information (as well as the interaction information) can be positive, negative, or zero, which makes this quantity difficult to interpret intuitively. In fact, for $\displaystyle{ n }$ random variables, there are $\displaystyle{ 2^n-1 }$ degrees of freedom for how they might be correlated in an information-theoretic sense, corresponding to each non-empty subset of these variables. These degrees of freedom are bounded by various Shannon- and non-Shannon-type inequalities in information theory.

## References 参考文献

1. Wyner, A. D. (1978). "A definition of conditional mutual information for arbitrary ensembles". Information and Control. 38 (1): 51–59. doi:10.1016/s0019-9958(78)90026-8.
2. Dobrushin, R. L. (1959). "General formulation of Shannon's main theorem in information theory". Uspekhi Mat. Nauk. 14: 3–104.
3. Cover, Thomas; Thomas, Joy A. (2006). Elements of Information Theory (2nd ed.). New York: Wiley-Interscience. ISBN 0-471-24195-4.
4. Cover, Thomas; Thomas, Joy A. (2006). Elements of Information Theory (2nd ed.). New York: Wiley-Interscience. ISBN 0-471-24195-4.
5. Decomposition on Math.StackExchange
6. Decomposition on Math.StackExchange
7. D. Leao, Jr. et al. Regular conditional probability, disintegration of probability and Radon spaces. Proyecciones. Vol. 23, No. 1, pp. 15–29, May 2004, Universidad Católica del Norte, Antofagasta, Chile PDF
8. D. Leao, Jr. et al. Regular conditional probability, disintegration of probability and Radon spaces. Proyecciones. Vol. 23, No. 1, pp. 15–29, May 2004, Universidad Católica del Norte, Antofagasta, Chile PDF
9. Polyanskiy, Yury; Wu, Yihong (2017). Lecture notes on information theory. p. 30.
10. Polyanskiy, Yury; Wu, Yihong (2017). Lecture notes on information theory. p. 30.