更改

[[File:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram]] showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the [[joint entropy]] <math>H(X,Y)</math>. The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] <math>H(X)</math>, with the red being the [[conditional entropy]] <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the [[mutual information]] <math>\operatorname{I}(X;Y)</math>.  这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的'''联合熵 Joint entropy'''<math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的'''独立熵 Individual entropy'''<math>H(X)</math>，红色（差集）部分表示X的'''条件熵 Conditional entropy'''<math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的'''独立熵 Individual entropy'''<math>H(Y)</math>，蓝色（差集）部分表示X的'''条件熵 Conditional entropy'''<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的互信息<math>\operatorname{I}(X;Y)</math>）。]]

[[File:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram]] showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the [[joint entropy]] <math>H(X,Y)</math>. The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] <math>H(X)</math>, with the red being the [[conditional entropy]] <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the [[mutual information]] <math>\operatorname{I}(X;Y)</math>.  这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。]]

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the joint entropy <math>H(X,Y)</math>. The circle on the left (red and violet) is the individual entropy <math>H(X)</math>, with the red being the conditional entropy <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the mutual information <math>\operatorname{I}(X;Y)</math>.

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the joint entropy <math>H(X,Y)</math>. The circle on the left (red and violet) is the individual entropy <math>H(X)</math>, with the red being the conditional entropy <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the mutual information <math>\operatorname{I}(X;Y)</math>.

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables 𝑋 and 𝑌. The area contained by both circles is the joint entropy H(𝑋,𝑌). The circle on the left (red and violet) is the individual entropy H(𝑋), with the red being the conditional entropy H(𝑋|𝑌). The circle on the right (blue and violet) is H(𝑌), with the blue being H(𝑌|𝑋). The violet is the mutual information I(𝑋;𝑌).

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables 𝑋 and 𝑌. The area contained by both circles is the joint entropy H(𝑋,𝑌). The circle on the left (red and violet) is the individual entropy H(𝑋), with the red being the conditional entropy H(𝑋|𝑌). The circle on the right (blue and violet) is H(𝑌), with the blue being H(𝑌|𝑋). The violet is the mutual information I(𝑋;𝑌).

这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information'''</font><math>\operatorname{I}(X;Y)</math>）。

这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable.  The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable.  The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

在<font color="#ff8000"> '''概率论 Probability theory'''</font>和'<font color="#ff8000"> ''信息论 Information theory'''</font>中，两个随机变量的<font color="#ff8000"> '''互信息 Mutual Information，MI'''</font>是两个变量之间相互依赖性的度量。更具体地说，它量化了通过观察另一个随机变量而获得的关于一个随机变量的“信息量”（单位如''香农 shannons''，通常称为比特）。互信息的概念与随机变量的熵有着错综复杂的联系，熵是信息理论中的一个基本概念，它量化了随机变量中所包含的预期“信息量”。

在<font color="#ff8000"> '''概率论 Probability theory'''</font>和<font color="#ff8000"> '''信息论 Information theory'''</font>理论中，两个随机变量的<font color="#ff8000"> '''互信息 Mutual Information，MI'''</font>是两个变量之间相互依赖程度的度量。更具体地说，它量化了通过观察一个随机变量而可以获得的关于另一个随机变量的“信息量”（单位如''香农 Shannons''，通常称为比特）。互信息的概念与随机变量的熵之间有着错综复杂的联系，熵是信息论中的一个基本概念，它量化了随机变量中所包含的预期“信息量”。

Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair <math>(X,Y)</math> is to the product of the marginal distributions of <math>X</math> and <math>Y</math>. MI is the expected value of the pointwise mutual information (PMI).

Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair <math>(X,Y)</math> is to the product of the marginal distributions of <math>X</math> and <math>Y</math>. MI is the expected value of the pointwise mutual information (PMI).

除了实值随机变量和线性依赖之类的的相关系数之外，互信息更为普遍，它决定了一对变量<math>(X,Y)</math>的联合分布以及 <math>X</math> 和 <math>Y</math> 的边际分布之积有多大的不同。互信息 是'''点态互信息指数 PMI'''的期望值。

不仅限于实值随机变量和线性依赖之类的的相关系数，互信息表示的关系其实更加普遍，它决定了一对变量<math>(X,Y)</math>的联合分布与<math>X</math>和<math>Y</math>的<font color="#ff8000">'''边缘分布 Marginal distributions'''</font>之积的不同程度。互信息是'''点互信息 Pointwise mutual information，PMI'''的期望值。

Mutual Information is also known as information gain.

Mutual Information is also known as information gain.

互信息也称为'''信息增益 Information gain'''。

互信息也称为<font color="#ff8000">'''信息增益 Information gain'''</font>。

Let <math>(X,Y)</math> be a pair of random variables with values over the space <math>\mathcal{X}\times\mathcal{Y}</math>. If their joint distribution is <math>P_{(X,Y)}</math> and the marginal distributions are <math>P_X</math> and <math>P_Y</math>, the mutual information is defined as

Let <math>(X,Y)</math> be a pair of random variables with values over the space <math>\mathcal{X}\times\mathcal{Y}</math>. If their joint distribution is <math>P_{(X,Y)}</math> and the marginal distributions are <math>P_X</math> and <math>P_Y</math>, the mutual information is defined as

设一对随机变量<math>(X,Y)</math>，参数空间为<math>\mathcal{X}\times\mathcal{Y}</math>。若它们之间的的联合概率分布为<math>P_{(X,Y)}</math>，边际分布分别为<math>P_X</math>和<math>P_Y</math>，则它们之间的互信息定义为：

设一对随机变量<math>(X,Y)</math>的参数空间为<math>\mathcal{X}\times\mathcal{Y}</math>。若它们之间的的联合概率分布为<math>P_{(X,Y)}</math>，边缘分布分别为<math>P_X</math>和<math>P_Y</math>，则它们之间的互信息定义为：

where <math>D_{\mathrm{KL}}</math> is the [[Kullback–Leibler divergence]].

where <math>D_{\mathrm{KL}}</math> is the [[Kullback–Leibler divergence]].

其中<math>D_{\mathrm{KL}}</math>表示Kullback-Leibler散度。

其中<math>D_{\mathrm{KL}}</math>表示<font color="#ff8000">'''相对熵 Relative entropy，又称Kullback-Leibler散度'''</font>。