463
个编辑
更改
→与Kullback-Leibler散度的关系 Relation to Kullback–Leibler divergence
For jointly discrete or jointly continuous pairs <math>(X,Y)</math>,
For jointly discrete or jointly continuous pairs <math>(X,Y)</math>,
对于联合离散或联合连续变量对（𝑋，𝑌）,
mutual information is the [[Kullback–Leibler divergence]] of the product of the [[marginal distribution]]s, <math>p_X \cdot p_Y</math>, from the [[joint distribution]] <math>p_{(X,Y)}</math>, that is,
mutual information is the [[Kullback–Leibler divergence]] of the product of the [[marginal distribution]]s, <math>p_X \cdot p_Y</math>, from the [[joint distribution]] <math>p_{(X,Y)}</math>, that is,
mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, <math>p_X \cdot p_Y</math>, from the joint distribution <math>p_{(X,Y)}</math>, that is,
mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, <math>p_X \cdot p_Y</math>, from the joint distribution <math>p_{(X,Y)}</math>, that is,
互信息是边际分布乘积的 Kullback-Leibler 散度，也就是联合分布数学 p {(x，y)} / math 的乘积,
互信息是边际分布乘积的 Kullback-Leibler 散度，也就是联合分布<math>p_{(X,Y)}</math>的乘积，即：
<math>
</math>
[[文件:MI pic6.png|居中|缩略图]]
Furthermore, let <math>p_{X|Y=y}(x) = p_{(X,Y)}(x,y) / p_Y(y)</math> be the conditional mass or density function. Then, we have the identity
Furthermore, let <math>p_{X|Y=y}(x) = p_{(X,Y)}(x,y) / p_Y(y)</math> be the conditional mass or density function. Then, we have the identity
进一步地，设<math>p_{X|Y=y}(x) = p_{(X,Y)}(x,y) / p_Y(y)</math>是条件质量或密度函数。那么，我们就有了''身份（这里翻译存疑）''：
<math>
</math>
[[文件:MI pic7.png|居中|缩略图]]
联合离散随机变量的证明如下:
联合离散随机变量的证明如下:
[[文件:MI pic8.png|居中|缩略图]]
Similarly this identity can be established for jointly continuous random variables.
Similarly this identity can be established for jointly continuous random variables.
Note that here the Kullback–Leibler divergence involves integration over the values of the random variable <math>X</math> only, and the expression <math>D_\text{KL}(p_{X|Y} \parallel p_X)</math> still denotes a random variable because <math>Y</math> is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution <math>p_X</math> of <math>X</math> from the conditional distribution <math>p_{X|Y}</math> of <math>X</math> given <math>Y</math>: the more different the distributions <math>p_{X|Y}</math> and <math>p_X</math> are on average, the greater the information gain.
Note that here the Kullback–Leibler divergence involves integration over the values of the random variable <math>X</math> only, and the expression <math>D_\text{KL}(p_{X|Y} \parallel p_X)</math> still denotes a random variable because <math>Y</math> is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution <math>p_X</math> of <math>X</math> from the conditional distribution <math>p_{X|Y}</math> of <math>X</math> given <math>Y</math>: the more different the distributions <math>p_{X|Y}</math> and <math>p_X</math> are on average, the greater the information gain.
因此，互信息也可以理解为X的单变量分布<math>p_X</math>与给定<math>Y</math>的<math>X</math>的条件分布<math>p_{X|Y}</math>的Kullback–Leibler散度的期望：平均分布<math>p_{X|Y}</math>和<math>p_X</math>的分布差异越大，信息增益越大。
=== 互信息的贝叶斯估计 Bayesian estimation of mutual information ===
=== 互信息的贝叶斯估计 Bayesian estimation of mutual information ===