条件互信息
此词条Jie翻译,由Flipped完成审校.
In probability theory, particularly information theory, the conditional mutual information[1][2] is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.
在 概率论 Probability theory中，特别是 信息论 Information theory中， 条件互信息 Conditional mutual information [1][2]的基本形式表示为当给定第三个变量的情况下两个随机变量间互信息的期望值。
Definition 定义
For random variables [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], and [math]\displaystyle{ Z }[/math] with support sets [math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] and [math]\displaystyle{ \mathcal{Z} }[/math], we define the conditional mutual information as
对于具有 支持集 Probability theory [math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] 和 [math]\displaystyle{ \mathcal{Z} }[/math]的随机变量[math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math]和 [math]\displaystyle{ Z }[/math]，我们将条件互信息定义为：
[math]\displaystyle{ I(X;Y|Z) = \int_\mathcal{Z} D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} ) dP_{Z} }[/math]
This may be written in terms of the expectation operator:
这可以用期望运算符来表示：
[math]\displaystyle{ I(X;Y|Z) = \mathbb{E}_Z [D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} )] }[/math].
Thus [math]\displaystyle{ I(X;Y|Z) }[/math] is the expected (with respect to [math]\displaystyle{ Z }[/math]) Kullback–Leibler divergence from the conditional joint distribution [math]\displaystyle{ P_{(X,Y)|Z} }[/math] to the product of the conditional marginals [math]\displaystyle{ P_{X|Z} }[/math] and [math]\displaystyle{ P_{Y|Z} }[/math]. Compare with the definition of mutual information.
因此，相较于互信息的定义，[math]\displaystyle{ I(X;Y|Z) }[/math]可以表达为期望的 KL散度 Kullback–Leibler divergence （相对于[math]\displaystyle{ Z }[/math]），即从条件联合分布[math]\displaystyle{ P_{(X,Y)|Z} }[/math]到条件边际[math]\displaystyle{ P_{X|Z} }[/math] 和 [math]\displaystyle{ P_{Y|Z} }[/math]的乘积。
In terms of pmf's for discrete distributions 关于离散分布的概率质量函数
For discrete random variables [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], and [math]\displaystyle{ Z }[/math] with support sets [math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] and [math]\displaystyle{ \mathcal{Z} }[/math], the conditional mutual information [math]\displaystyle{ I(X;Y|Z) }[/math] is as follows
对于具有支持集[math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], 和 [math]\displaystyle{ Z }[/math]的离散随机变量[math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] 和 [math]\displaystyle{ \mathcal{Z} }[/math]，条件互信息[math]\displaystyle{ I(X;Y|Z) }[/math]如下:
- [math]\displaystyle{ I(X;Y|Z) = \sum_{z\in \mathcal{Z}} p_Z(z) \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} p_{X,Y|Z}(x,y|z) \log \frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)} }[/math]
where the marginal, joint, and/or conditional probability mass functions are denoted by [math]\displaystyle{ p }[/math] with the appropriate subscript. This can be simplified as
其中边缘概率质量函数，联合概率质量函数，和（或）条件概率质量函数 Probability mass function 可以由[math]\displaystyle{ p }[/math]加上适当的下标表示。这可以简化为:
[math]\displaystyle{ I(X;Y|Z) = \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} p_{X,Y,Z}(x,y,z) \log \frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}. }[/math]
In terms of pdf's for continuous distributions 关于连续分布的概率密度函数
For (absolutely) continuous random variables [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], and [math]\displaystyle{ Z }[/math] with support sets [math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] and [math]\displaystyle{ \mathcal{Z} }[/math], the conditional mutual information [math]\displaystyle{ I(X;Y|Z) }[/math] is as follows
对于具有支持集[math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], 和 [math]\displaystyle{ Z }[/math]的（绝对）连续随机变量[math]\displaystyle{ \mathcal{X} }[/math], [math]\displaystyle{ \mathcal{Y} }[/math] 和 [math]\displaystyle{ \mathcal{Z} }[/math]，条件互信息[math]\displaystyle{ I(X;Y|Z) }[/math]如下:
- [math]\displaystyle{ I(X;Y|Z) = \int_{\mathcal{Z}} \bigg( \int_{\mathcal{Y}} \int_{\mathcal{X}} \log \left(\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)}\right) p_{X,Y|Z}(x,y|z) dx dy \bigg) p_Z(z) dz }[/math]
where the marginal, joint, and/or conditional probability density functions are denoted by [math]\displaystyle{ p }[/math] with the appropriate subscript. This can be simplified as
其中边缘概率密度函数，联合概率密度函数，和（或）条件概率密度函数 Probability density function 可以由p加上适当的下标表示。这可以简化为
[math]\displaystyle{ I(X;Y|Z) = \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}} \log \left(\frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}\right) p_{X,Y,Z}(x,y,z) dx dy dz. }[/math]
Some identities 部分特性
Alternatively, we may write in terms of joint and conditional entropies as[3]
同时我们也可以将联合和条件熵写为[4]：
- [math]\displaystyle{ I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z) = H(X|Z) - H(X|Y,Z) = H(X|Z)+H(Y|Z)-H(X,Y|Z). }[/math]
This can be rewritten to show its relationship to mutual information
这么表达以显示其与互信息的关系
- [math]\displaystyle{ I(X;Y|Z) = I(X;Y,Z) - I(X;Z) }[/math]
usually rearranged as the chain rule for mutual information
通常情况下，表达式被重新整理为“互信息的链式法则”
- [math]\displaystyle{ I(X;Y,Z) = I(X;Z) + I(X;Y|Z) }[/math]
Another equivalent form of the above is[5]
上述式子的另一种等价形式是[6]：
- [math]\displaystyle{ I(X;Y|Z) = H(Z|X) + H(X) + H(Z|Y) + H(Y) - H(Z|X,Y) - H(X,Y) - H(Z) = I(X;Y) + H(Z|X) + H(Z|Y) - H(Z|X,Y) - H(Z) }[/math]
Like mutual information, conditional mutual information can be expressed as a Kullback–Leibler divergence:
类似互信息一样，条件互信息可以表示为Kullback-Leibler散度：
- [math]\displaystyle{ I(X;Y|Z) = D_{\mathrm{KL}}[ p(X,Y,Z) \| p(X|Z)p(Y|Z)p(Z) ]. }[/math]
Or as an expected value of simpler Kullback–Leibler divergences:
或作为更简单的Kullback-Leibler散度的期望值：
- [math]\displaystyle{ I(X;Y|Z) = \sum_{z \in \mathcal{Z}} p( Z=z ) D_{\mathrm{KL}}[ p(X,Y|z) \| p(X|z)p(Y|z) ] }[/math],
- [math]\displaystyle{ I(X;Y|Z) = \sum_{y \in \mathcal{Y}} p( Y=y ) D_{\mathrm{KL}}[ p(X,Z|y) \| p(X|Z)p(Z|y) ] }[/math].
More general definition 其他通用定义
A more general definition of conditional mutual information, applicable to random variables with continuous or other arbitrary distributions, will depend on the concept of regular conditional probability. (See also. [7][8])
条件互信息的其他通用定义（适用于具有连续或其他任意分布的随机变量）将取决于 正则条件概率 Regular conditional probability 的概念。(参阅[9][10]))
Let [math]\displaystyle{ (\Omega, \mathcal F, \mathfrak P) }[/math] be a probability space, and let the random variables [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math], and [math]\displaystyle{ Z }[/math] each be defined as a Borel-measurable function from [math]\displaystyle{ \Omega }[/math] to some state space endowed with a topological structure.
令[math]\displaystyle{ (\Omega, \mathcal F, \mathfrak P) }[/math]为一个 概率空间 Probability space ，并将随机变量[math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math]和 [math]\displaystyle{ Z }[/math]分别定义为一个从[math]\displaystyle{ \Omega }[/math]到具有拓扑结构的状态空间的 波莱尔可测函数 Borel-measurable function 。
Consider the Borel measure (on the σ-algebra generated by the open sets) in the state space of each random variable defined by assigning each Borel set the [math]\displaystyle{ \mathfrak P }[/math]-measure of its preimage in [math]\displaystyle{ \mathcal F }[/math]. This is called the pushforward measure [math]\displaystyle{ X _* \mathfrak P = \mathfrak P\big(X^{-1}(\cdot)\big). }[/math] The support of a random variable is defined to be the topological support of this measure, i.e. [math]\displaystyle{ \mathrm{supp}\,X = \mathrm{supp}\,X _* \mathfrak P. }[/math]
考虑到在每个随机变量状态空间中的 波莱尔测度 Borel measure（关于开放集生成的σ代数），是由[math]\displaystyle{ \mathcal F }[/math]中每个波莱尔集分配到的的原像[math]\displaystyle{ \mathfrak P }[/math]测度来确定的。这被称为 前推测度 Pushforward measure [math]\displaystyle{ X _* \mathfrak P = \mathfrak P\big(X^{-1}(\cdot)\big). }[/math]。随机变量的支撑集定义为该测度的拓扑支撑集，即[math]\displaystyle{ \mathrm{supp}\,X = \mathrm{supp}\,X _* \mathfrak P. }[/math]。
Now we can formally define the conditional probability measure given the value of one (or, via the product topology, more) of the random variables. Let [math]\displaystyle{ M }[/math] be a measurable subset of [math]\displaystyle{ \Omega, }[/math] (i.e. [math]\displaystyle{ M \in \mathcal F, }[/math]) and let [math]\displaystyle{ x \in \mathrm{supp}\,X. }[/math] Then, using the disintegration theorem:
现在，我们可以在给定其中一个随机变量值（或通过 积拓扑 product topology 获得更多）的情况下正式定义 条件概率测度 Conditional probability distribution|conditional probability measure 。令[math]\displaystyle{ M }[/math]为[math]\displaystyle{ \Omega }[/math]的可测子集（即[math]\displaystyle{ M \in \mathcal F, }[/math]），令[math]\displaystyle{ x \in \mathrm{supp}\,X }[/math]。然后，使用 分解定理 Disintegration theorem ：
- [math]\displaystyle{ \mathfrak P(M | X=x) = \lim_{U \ni x} \frac {\mathfrak P(M \cap \{X \in U\})} {\mathfrak P(\{X \in U\})} \qquad \textrm{and} \qquad \mathfrak P(M|X) = \int_M d\mathfrak P\big(\omega|X=X(\omega)\big), }[/math]
where the limit is taken over the open neighborhoods [math]\displaystyle{ U }[/math] of [math]\displaystyle{ x }[/math], as they are allowed to become arbitrarily smaller with respect to set inclusion.
在[math]\displaystyle{ x }[/math]的开放邻域[math]\displaystyle{ U }[/math]处取极限，因为相对于 集包含 Set inclusion，它们可以任意变小。
Finally we can define the conditional mutual information via Lebesgue integration:
最后，我们可以通过 勒贝格积分 Lebesgue integration来定义条件互信息：
- [math]\displaystyle{ I(X;Y|Z) = \int_\Omega \log \Bigl( \frac {d \mathfrak P(\omega|X,Z)\, d\mathfrak P(\omega|Y,Z)} {d \mathfrak P(\omega|Z)\, d\mathfrak P(\omega|X,Y,Z)} \Bigr) d \mathfrak P(\omega), }[/math]
where the integrand is the logarithm of a Radon–Nikodym derivative involving some of the conditional probability measures we have just defined.
其中被积函数是 拉东-尼科迪姆导数 Radon–Nikodym derivative的对数，涉及我们刚刚定义的一些条件概率测度。
Note on notation 注释符号
In an expression such as [math]\displaystyle{ I(A;B|C), }[/math] [math]\displaystyle{ A, }[/math] [math]\displaystyle{ B, }[/math] and [math]\displaystyle{ C }[/math] need not necessarily be restricted to representing individual random variables, but could also represent the joint distribution of any collection of random variables defined on the same probability space. As is common in probability theory, we may use the comma to denote such a joint distribution, e.g. [math]\displaystyle{ I(A_0,A_1;B_1,B_2,B_3|C_0,C_1). }[/math] Hence the use of the semicolon (or occasionally a colon or even a wedge [math]\displaystyle{ \wedge }[/math]) to separate the principal arguments of the mutual information symbol. (No such distinction is necessary in the symbol for joint entropy, since the joint entropy of any number of random variables is the same as the entropy of their joint distribution.)
在诸如[math]\displaystyle{ I(A;B|C) }[/math]的表达式中，[math]\displaystyle{ A }[/math] [math]\displaystyle{ B }[/math] 和 [math]\displaystyle{ C }[/math]不限于表示单个随机变量，它们同时可以表示在同一概率空间上定义的任意随机变量集合的联合分布。类似概率论中的表达方式，我们可以使用逗号来表示这种联合分布，例如[math]\displaystyle{ I(A_0,A_1;B_1,B_2,B_3|C_0,C_1) }[/math]。因此，使用分号（或有时用冒号或楔形[math]\displaystyle{ \wedge }[/math]）来分隔互信息符号的主要参数。（在联合熵的符号中，不需要作这样的区分，因为任意数量随机变量的 联合熵 Joint entropy与它们联合分布的熵相同。）
Properties 属性
Nonnegativity 非负性
It is always true that
- [math]\displaystyle{ I(X;Y|Z) \ge 0 }[/math],
for discrete, jointly distributed random variables [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math] and [math]\displaystyle{ Z }[/math]. This result has been used as a basic building block for proving other inequalities in information theory, in particular, those known as Shannon-type inequalities. Conditional mutual information is also non-negative for continuous random variables under certain regularity conditions.[11]
对于离散，联合分布的随机变量X，Y和Z，如下不等式永远成立：
- [math]\displaystyle{ I(X;Y|Z) \ge 0 }[/math],
该结果已被用作证明信息理论中其他不等式的基础，尤其是香农不等式。对于某些正则条件下的连续随机变量，条件互信息也是非负的[12]。
Interaction information 交互信息
Conditioning on a third random variable may either increase or decrease the mutual information: that is, the difference [math]\displaystyle{ I(X;Y) - I(X;Y|Z) }[/math], called the interaction information, may be positive, negative, or zero. This is the case even when random variables are pairwise independent. Such is the case when:
[math]\displaystyle{ X \sim \mathrm{Bernoulli}(0.5), Z \sim \mathrm{Bernoulli}(0.5), \quad Y=\left\{\begin{array}{ll} X & \text{if }Z=0\\ 1-X & \text{if }Z=1 \end{array}\right. }[/math]
in which case [math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math] and [math]\displaystyle{ Z }[/math] are pairwise independent and in particular [math]\displaystyle{ I(X;Y)=0 }[/math], but [math]\displaystyle{ I(X;Y|Z)=1. }[/math]
考虑到第三个随机变量条件可能会增加或减少 互信息 Mutual information ：例如其差值[math]\displaystyle{ I(X;Y) - I(X;Y|Z) }[/math]，称为 交互信息 Interaction information (注意区分互信息Mutual information)，可以为正，负或零。即使随机变量是成对独立的也是如此。比如以下情况下：
[math]\displaystyle{ X \sim \mathrm{Bernoulli}(0.5), Z \sim \mathrm{Bernoulli}(0.5), \quad Y=\left\{\begin{array}{ll} X & \text{if }Z=0\\ 1-X & \text{if }Z=1 \end{array}\right. }[/math]
[math]\displaystyle{ X }[/math], [math]\displaystyle{ Y }[/math] 和 [math]\displaystyle{ Z }[/math]是成对独立的，特别是[math]\displaystyle{ I(X;Y)=0 }[/math]，不过这里[math]\displaystyle{ I(X;Y|Z)=1. }[/math]。
Chain rule for mutual information 互信息的链式法则
- [math]\displaystyle{ I(X;Y,Z) = I(X;Z) + I(X;Y|Z) }[/math]
Multivariate mutual information 多元互信息
The conditional mutual information can be used to inductively define a multivariate mutual information in a set- or measure-theoretic sense in the context of information diagrams. In this sense we define the multivariate mutual information as follows:
结合信息图中的集合或度量理论，可以用条件互信息来归纳定义多元互信息。其定义表达式如下：
- [math]\displaystyle{ I(X_1;\ldots;X_{n+1}) = I(X_1;\ldots;X_n) - I(X_1;\ldots;X_n|X_{n+1}), }[/math]
Where
其中
- [math]\displaystyle{ I(X_1;\ldots;X_n|X_{n+1}) = \mathbb{E}_{X_{n+1}} [D_{\mathrm{KL}}( P_{(X_1,\ldots,X_n)|X_{n+1}} \| P_{X_1|X_{n+1}} \otimes\cdots\otimes P_{X_n|X_{n+1}} )]. }[/math]
This definition is identical to that of interaction information except for a change in sign in the case of an odd number of random variables. A complication is that this multivariate mutual information (as well as the interaction information) can be positive, negative, or zero, which makes this quantity difficult to interpret intuitively. In fact, for [math]\displaystyle{ n }[/math] random variables, there are [math]\displaystyle{ 2^n-1 }[/math] degrees of freedom for how they might be correlated in an information-theoretic sense, corresponding to each non-empty subset of these variables. These degrees of freedom are bounded by various Shannon- and non-Shannon-type inequalities in information theory.
该定义与交互信息的定义相同，只是在随机数为奇数的情况下符号发生了变化。一个复杂的问题是，该多元互信息（以及交互信息）可以是正，负或零，这使得其数量难以直观地解释。实际上，对于n个随机变量，存在2n-1个自由度。那么如何在信息理论上将它们关联，并对应于这些变量的每个非空子集，就是解决问题的关键。特别是这些自由度受到信息论中各种香农和非香农不等式的制约。
References 参考文献
- ↑ 1.0 1.1 Wyner, A. D. (1978). "A definition of conditional mutual information for arbitrary ensembles". Information and Control. 38 (1): 51–59. doi:10.1016/s0019-9958(78)90026-8.
- ↑ 2.0 2.1 Dobrushin, R. L. (1959). "General formulation of Shannon's main theorem in information theory". Uspekhi Mat. Nauk. 14: 3–104.
- ↑ Cover, Thomas; Thomas, Joy A. (2006). Elements of Information Theory (2nd ed.). New York: Wiley-Interscience. ISBN 0-471-24195-4.
- ↑ Cover, Thomas; Thomas, Joy A. (2006). Elements of Information Theory (2nd ed.). New York: Wiley-Interscience. ISBN 0-471-24195-4.
- ↑ Decomposition on Math.StackExchange
- ↑ Decomposition on Math.StackExchange
- ↑ Regular Conditional Probability on PlanetMath
- ↑ D. Leao, Jr. et al. Regular conditional probability, disintegration of probability and Radon spaces. Proyecciones. Vol. 23, No. 1, pp. 15–29, May 2004, Universidad Católica del Norte, Antofagasta, Chile PDF
- ↑ Regular Conditional Probability on PlanetMath
- ↑ D. Leao, Jr. et al. Regular conditional probability, disintegration of probability and Radon spaces. Proyecciones. Vol. 23, No. 1, pp. 15–29, May 2004, Universidad Católica del Norte, Antofagasta, Chile PDF
- ↑ Polyanskiy, Yury; Wu, Yihong (2017). Lecture notes on information theory. p. 30. http://people.lids.mit.edu/yp/homepage/data/itlectures_v5.pdf.
- ↑ Polyanskiy, Yury; Wu, Yihong (2017). Lecture notes on information theory. p. 30. http://people.lids.mit.edu/yp/homepage/data/itlectures_v5.pdf.