463
个编辑
更改
跳到导航
跳到搜索
第1行：
第1行： − 此词条暂由彩云小译翻译，未经人工整理和审校，带来阅读不便，请见谅。{{Information theory}}+ 第7行：
第7行： − + + 第15行：
第16行： − + 第23行：
第24行： − + 第33行：
第34行： − + 第43行：
第44行： − + 第64行：
第65行： − + 第70行：
第71行： − 需要注意的是，根据Kullback–Leibler散度的性质，当两个随机变量的联合分布与其分别的边缘分布的乘积相等时，即当<math>X</math>和<math>Y</math>是相互独立的时，<math>I(X;Y)</math>等于零（因此已知<math>Y</math>的信息并不能得到任何关于<math>X</math>的信息）。'''<font color="#32CD32">一般来说，<math>I(X;Y)</math>是非负的，因为它是将<math>(X,Y)</math>作为一对独立随机变量来编码进而来进行价格（价值）度量的，但实际上它们并不一定是非负的。</font>'''+ 第87行：
第88行： − + 第105行：
第106行： − + 第123行：
第124行： − 直观地说，相互信息衡量了<math>X</math> and <math>Y</math>的信息共享程度：它衡量了当已知其中一个变量后可以减少另一个变量多少的不确定性。例如，若<math>X</math>和<math>Y</math>是相互独立的，那么已知<math>X</math>不会得到关于<math>Y</math>的任何信息，反之亦然，因此它们之间的互信息为零。而另一种极端情况就是，若<math>X</math>是<math>Y</math>的'''<font color="#32CD32">确定函数</font>'''，而<math>X</math>也是<math>X</math>自身的确定函数，则<math>X</math>传递的所有信息都与<math>Y</math>共享：即已知<math>X</math>就可以知道<math>Y</math>的值，反之亦然。因此，在这种情况下，互信息与仅包含在<math>Y</math>（或<math>X</math>）中的不确定性相同，即<math>Y</math>（或<math>X</math>）的熵相同。此外，这种情况下互信息与<math>X</math>的熵和<math>Y</math>的熵相同。（一个非常特殊的情况是当<math>X</math>和<math>Y</math>是相同的随机变量。）+ 第158行：
第159行： − + 第181行：
第182行： − + 第250行：
第251行： − 互信息是边缘分布乘积的相对熵 <math>D_{KL}</math>，也就是联合分布<math>p_{(X,Y)}</math>的乘积，即：+ 第290行：
第291行： − + 第326行：
第327行： − 相互信息的Kullback-Leibler散度公式是基于这样一个结论的：人们会更关注将<math>p(x,y)</math>与完全分解的外积<math>p(x) \cdot p(y)</math>进行比较。在许多问题中，例如非负矩阵因式分解中，人们对'''<font color="#32CD32">较不极端的</font>'''因式分解感兴趣；具体地说，人们希望将<math>p(x,y)</math>与某个未知变量<math>w</math>中的低秩矩阵近似进行比较；也就是说，在多大程度上可能会有这样的结果：+ 第336行：
第337行： − + 第441行：
第442行： − + 第590行：
第591行： − + 第661行：
第662行： − + 第672行：
第673行： − + 第678行：
第679行： − + 第686行：
第687行： − + 第699行：
第700行： − + 第748行：
第749行： − 当一个变量与另一个变量的知识完全多余时。参见冗余(信息论)。+ 第758行：
第759行： − 另一个对称度量是对称不确定度，由下式表示：+ 第771行：
第772行： − 它表示两个不确定系数的调和平均数。+ 第781行：
第782行： − + 第791行：
第792行： − + 第808行：
第809行： − + 第875行：
第876行： − + 第883行：
第884行： − + 第901行：
第902行： − + 第912行：
第913行： − 互信息不同于相关系数，如乘积矩相关系数，互信息包含所有相关信息ーー线性和非线性ーー而不仅仅是相关系数的线性相关。然而，在数学 x / math 和数学 y / math 的联合分布是二元正态分布(特别是边际分布都是正态分布)的狭义情况下，数学运算子{ i } / math 与相关系数 math / rho / math 之间存在精确的关系。+ 第978行：
第979行： − + 第1,012行：
第1,013行： − + 第1,026行：
第1,027行： − 现在已经提出了基于最大互信息（MMI）准则的隐马尔可夫模型判别训练方法。+ 第1,052行：
第1,053行： − 互信息用于确定数据集中两个不同聚类的相似性。因此，它与传统的Rand指数相比具有一定的优势。+ 第1,058行：
第1,059行： − 在语料库语言学中，词的互信息常常被用作计算搭配的意义函数。这增加了复杂性，即没有一个单词实例是两个不同单词的实例；相反，我们统计两个单词相邻或非常接近的实例；这稍微使计算复杂化，因为一个单词出现在另一个单词的𝑁单词中的预期概率会增加。+ 第1,064行：
第1,065行： − 在医学成像中，利用互信息进行图像配准。给定一个参考图像（例如，脑部扫描），以及需要将第二个图像放入与参考图像相同的坐标系中，该图像会发生变形，直到其与参考图像之间的互信息最大化。+ 第1,070行：
第1,071行： − 时间序列分析中的相位同步检测。+ 第1,076行：
第1,077行： − 在infomax方法中用于神经网络等机器学习，包括基于infomax的独立分量分析算法+ 第1,082行：
第1,083行： − 利用延迟嵌入定理中的平均互信息确定嵌入延迟参数。+ 第1,088行：
第1,089行： − ARACNE算法利用表达微阵列数据中基因间的互信息来重构基因网络。+ 第1,095行：
第1,096行： − 在统计力学中，Loschmidt悖论可以用相互信息来表示。Loschmidt指出，只有从具有这种对称性的物理定律中确定缺乏时间反转对称性的物理定律（例如热力学第二定律）是不可能的。他指出，玻尔兹曼的H-定理假设气体中粒子的速度是永久不相关的，这就消除了H-定理固有的时间对称性。可以证明，如果系统在相空间中用概率密度来描述，那么Liouville定理意味着分布的联合信息（联合熵的负）在时间上保持不变。关节信息等于互信息加上每个粒子坐标的所有边缘信息（负的边缘熵）之和。玻尔兹曼的假设相当于在熵的计算中忽略了相互信息，从而得到了热力学熵（除以玻尔兹曼常数）。+ 第1,102行：
第1,103行： − 互信息用于学习贝叶斯网络/动态贝叶斯网络的结构，被认为是用来解释随机变量之间的因果关系，如GlobalMIT工具包[29]用互信息检验准则学习全局最优动态贝叶斯网络。+ 第1,109行：
第1,110行： − 作为决策树学习中常用的代价函数。+ 第1,115行：
第1,116行： − + 第1,122行：
第1,123行： − 在太阳物理学中，相互信息被用于推导太阳差分自转剖面图、太阳黑子的旅行时间偏差图和安静太阳测量的时间-距离图。+ 第1,132行：
第1,133行： − + + − + − 第1,363行：
第1,364行： − − − − | last1 = Wells − − − 1 Wells − − | first1 = W.M. III − − − 首先是 w.m。三 − − | last2 = Viola − − − 2 Viola − − | first2 = P. − − | first2 p. − − | last3 = Atsumi − − − 3 Atsumi − − | first3 = H. − − − 第一个3 h。 − − | last4 = Nakajima − − | 最后4名中岛 − − | first4 = S. − − 4 s. − − | last5 = Kikinis − − 最后5个 Kikinis − − | first5 = R. − − − | first5 r. − − | title = Multi-modal volume registration by maximization of mutual information − − − 最大化互信息的多模态卷注册 − − | journal = Medical Image Analysis − − 医学图像分析 − − − | volume = 1 − − 第一卷 − − | issue = 1 − − − 第一期 − − | pages = 35–51 − − − 第35-51页 − − | year = 1996 − − 1996年 − − | pmid = 9873920 − − 9873920 − − | doi = 10.1016/S1361-8415(01)80004-9 − − | doi 10.1016 / S1361-8415(01)80004-9 − − | url = http://www.ai.mit.edu/people/sw/papers/mia.pdf − − Http://www.ai.mit.edu/people/sw/papers/mia.pdf − − − | ref = harv − − 不会有事的 − − | access-date = 2010-08-05 − − 2010-08-05 − − | archive-url = https://web.archive.org/web/20080906201633/http://www.ai.mit.edu/people/sw/papers/mia.pdf − − − | 档案-网址 https://web.archive.org/web/20080906201633/http://www.ai.mit.edu/people/sw/papers/mia.pdf − − | archive-date = 2008-09-06 − − 2008-09-06 − − | url-status = dead − − − 状态死机 − − }} − − }} −
无编辑摘要
已由[[Xebec]]进行初步翻译。{{Information theory}}
[[File:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram]] showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the [[joint entropy]]
[[File:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram]] showing additive and subtractive relationships various information measures associated with correlated variables <math>X</math> and <math>Y</math>. The area contained by both circles is the [[joint entropy 这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。]]
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）图片应该按照[图1：英文+中文]
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）图片应该按照[图1：英文+中文]
<math>H(X,Y)</math>. The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] <math>H(X)</math>, with the red being the [[conditional entropy]] <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the [[mutual information]] <math>\operatorname{I}(X;Y)</math>. 这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。]]
<math>H(X,Y)</math>. The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] <math>H(X)</math>, with the red being the [[conditional entropy]] <math>H(X|Y)</math>. The circle on the right (blue and violet) is <math>H(Y)</math>, with the blue being <math>H(Y|X)</math>. The violet is the [[mutual information]] <math>\operatorname{I}(X;Y)</math>. 这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。]]
Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables 𝑋 and 𝑌. The area contained by both circles is the joint entropy H(𝑋,𝑌). The circle on the left (red and violet) is the individual entropy H(𝑋), with the red being the conditional entropy H(𝑋|𝑌). The circle on the right (blue and violet) is H(𝑌), with the blue being H(𝑌|𝑋). The violet is the mutual information I(𝑋;𝑌).
Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables 𝑋 and 𝑌. The area contained by both circles is the joint entropy H(𝑋,𝑌). The circle on the left (red and violet) is the individual entropy H(𝑋), with the red being the conditional entropy H(𝑋|𝑌). The circle on the right (blue and violet) is H(𝑌), with the blue being H(𝑌|𝑋). The violet is the mutual information I(𝑋;𝑌).
这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。
这里的维恩图显示了各种信息间的交并补运算关系关系，这些信息都可以用来度量变量<math>X</math>和<math>Y</math>的各种相关性。图中所有面积（包括两个圆圈）表示二者的<font color="#ff8000"> '''联合熵 Joint Entropy'''</font><math>H(X,Y)</math>。左侧的整个圆圈表示变量<math>X</math>的<font color="#ff8000"> '''独立熵 Individual Entropy'''</font><math>H(X)</math>，红色（差集）部分表示X的<font color="#ff8000"> '''条件熵 Conditional Entropy'''</font><math>H(X|Y)</math>。右侧的整个圆圈表示变量<math>Y</math>的独立熵<math>H(Y)</math>，蓝色（差集）部分表示X的条件熵<math>H(Y|X)</math>。两个圆中间的交集部分（紫色的部分）表示二者的<font color="#ff8000">'''互信息 Mutual Information，MI'''</font><math>\operatorname{I}(X;Y)</math>）。
In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.
In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.
在<font color="#ff8000"> '''概率论 Probability theory'''</font>和<font color="#ff8000"> '''信息论 Information theory'''</font>理论中，两个随机变量的<font color="#ff8000"> '''互信息 Mutual Information，MI'''</font>是两个变量之间相互依赖程度的度量。更具体地说，它量化了通过观察一个随机变量而可以获得的关于另一个随机变量的“信息量”（单位如''香农 Shannons''，通常称为比特）。互信息的概念与随机变量的熵之间有着错综复杂的联系，熵是信息论中的一个基本概念，它量化了随机变量中所包含的预期“信息量”。
在<font color="#ff8000"> '''概率论 Probability Theory'''</font>和<font color="#ff8000"> '''信息论 Information Theory'''</font>理论中，两个随机变量的<font color="#ff8000"> '''互信息 Mutual Information，MI'''</font>是两个变量之间相互依赖程度的度量。更具体地说，它量化了通过观察一个随机变量而可以获得的关于另一个随机变量的“信息量”（单位如''香农 Shannons''，通常称为比特）。互信息的概念与随机变量的熵之间有着错综复杂的联系，熵是信息论中的一个基本概念，它量化了随机变量中所包含的预期“信息量”。
Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair <math>(X,Y)</math> is to the product of the marginal distributions of <math>X</math> and <math>Y</math>. MI is the expected value of the pointwise mutual information (PMI).
Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair <math>(X,Y)</math> is to the product of the marginal distributions of <math>X</math> and <math>Y</math>. MI is the expected value of the pointwise mutual information (PMI).
不仅限于实值随机变量和线性依赖之类的的相关系数，互信息表示的关系其实更加普遍，它决定了一对变量<math>(X,Y)</math>的联合分布与<math>X</math>和<math>Y</math>的<font color="#ff8000">'''边缘分布 Marginal distributions'''</font>之积的不同程度。互信息是'''点互信息 Pointwise mutual information，PMI'''的期望值。
不仅限于实值随机变量和线性依赖之类的的相关系数，互信息表示的关系其实更加普遍，它决定了一对变量<math>(X,Y)</math>的联合分布与<math>X</math>和<math>Y</math>的<font color="#ff8000">'''边缘分布 Marginal Distributions'''</font>之积的不同程度。互信息是'''点互信息 Pointwise Mutual Information，PMI'''的期望值。
Mutual Information is also known as information gain.
Mutual Information is also known as information gain.
互信息也称为<font color="#ff8000">'''信息增益 Information gain'''</font>。
互信息也称为<font color="#ff8000">'''信息增益 Information Gain'''</font>。
where <math>D_{\mathrm{KL}}</math> is the [[Kullback–Leibler divergence]].
where <math>D_{\mathrm{KL}}</math> is the [[Kullback–Leibler divergence]].
其中<math>D_{\mathrm{KL}}</math>表示<font color="#ff8000">'''相对熵 Relative entropy，又称Kullback-Leibler散度'''</font>。
其中<math>D_{\mathrm{KL}}</math>表示<font color="#ff8000">'''相对熵 Relative Entropy，又称Kullback-Leibler/KL散度'''（以下统称KL散度）</font>。
Notice, as per property of the [[Kullback–Leibler divergence]], that <math>I(X;Y)</math> is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when <math>X</math> and <math>Y</math> are independent (and hence observing <math>Y</math> tells you nothing about <math>X</math>). '''<font color="#32CD32">In general <math>I(X;Y)</math> is non-negative, it is a measure of the price for encoding <math>(X,Y)</math> as a pair of independent random variables, when in reality they are not.</font>'''
Notice, as per property of the [[Kullback–Leibler divergence]], that <math>I(X;Y)</math> is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when <math>X</math> and <math>Y</math> are independent (and hence observing <math>Y</math> tells you nothing about <math>X</math>). '''<font color="#32CD32">In general <math>I(X;Y)</math> is non-negative, it is a measure of the price for encoding <math>(X,Y)</math> as a pair of independent random variables, when in reality they are not.</font>'''
需要注意的是，根据KL散度的性质，当两个随机变量的联合分布与其分别的边缘分布的乘积相等时，即当<math>X</math>和<math>Y</math>是相互独立的时，<math>I(X;Y)</math>等于零（因此已知<math>Y</math>的信息并不能得到任何关于<math>X</math>的信息）。'''<font color="#32CD32">一般来说，<math>I(X;Y)</math>是非负的，因为它是将<math>(X,Y)</math>作为一对独立随机变量来编码进而来进行价格（价值）度量的，但实际上它们并不一定是非负的。</font>'''
== 关于离散分布的PMF In terms of PMFs for discrete distributions ==
== 关于离散分布的PMF In terms of PMFs for discrete distributions ==
where <math>p_{(X,Y)}</math> is the joint probability mass function of <math>X</math> and <math>Y</math>, and <math>p_X</math> and <math>p_Y</math> are the marginal probability mass functions of <math>X</math> and <math>Y</math> respectively.
where <math>p_{(X,Y)}</math> is the joint probability mass function of <math>X</math> and <math>Y</math>, and <math>p_X</math> and <math>p_Y</math> are the marginal probability mass functions of <math>X</math> and <math>Y</math> respectively.
<math>p_{(X,Y)}</math>是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">概率质量函数 Probability mass functions</font>'''，而<math>p_X</math>和<math>p_Y</math>分别是数学<math>X</math>和<math>Y</math>的'''<font color="#ff8000">边缘概率质量函数 Marginal probability mass functions</font>'''。
<math>p_{(X,Y)}</math>是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">概率质量函数 Probability Mass Functions</font>'''，而<math>p_X</math>和<math>p_Y</math>分别是数学<math>X</math>和<math>Y</math>的'''<font color="#ff8000">边缘概率质量函数 Marginal Probability Mass Functions</font>'''。
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）注意首字母大写
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）注意首字母大写
where <math>p_{(X,Y)}</math> is now the joint probability density function of <math>X</math> and <math>Y</math>, and <math>p_X</math> and <math>p_Y</math> are the marginal probability density functions of <math>X</math> and <math>Y</math> respectively.
where <math>p_{(X,Y)}</math> is now the joint probability density function of <math>X</math> and <math>Y</math>, and <math>p_X</math> and <math>p_Y</math> are the marginal probability density functions of <math>X</math> and <math>Y</math> respectively.
式中，<math>p_{(X,Y)}</math>是<math>X</math>和<math>Y</math>的联合概率密度函数，而<math>p_X</math>和<math>p_Y</math>分别是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">边缘概率密度函数 Probability density function</font>'''。
式中，<math>p_{(X,Y)}</math>是<math>X</math>和<math>Y</math>的联合概率密度函数，而<math>p_X</math>和<math>p_Y</math>分别是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">边缘概率密度函数 Probability Density Function</font>'''。
Intuitively, mutual information measures the information that <math>X</math> and <math>Y</math> share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if <math>X</math> and <math>Y</math> are independent, then knowing <math>X</math> does not give any information about <math>Y</math> and vice versa, so their mutual information is zero. At the other extreme, if <math>X</math> is a deterministic function of <math>Y</math> and <math>Y</math> is a deterministic function of <math>X</math> then all information conveyed by <math>X</math> is shared with <math>Y</math>: knowing <math>X</math> determines the value of <math>Y</math> and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in <math>Y</math> (or <math>X</math>) alone, namely the entropy of <math>Y</math> (or <math>X</math>). Moreover, this mutual information is the same as the entropy of <math>X</math> and as the entropy of <math>Y</math>. (A very special case of this is when <math>X</math> and <math>Y</math> are the same random variable.)
Intuitively, mutual information measures the information that <math>X</math> and <math>Y</math> share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if <math>X</math> and <math>Y</math> are independent, then knowing <math>X</math> does not give any information about <math>Y</math> and vice versa, so their mutual information is zero. At the other extreme, if <math>X</math> is a deterministic function of <math>Y</math> and <math>Y</math> is a deterministic function of <math>X</math> then all information conveyed by <math>X</math> is shared with <math>Y</math>: knowing <math>X</math> determines the value of <math>Y</math> and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in <math>Y</math> (or <math>X</math>) alone, namely the entropy of <math>Y</math> (or <math>X</math>). Moreover, this mutual information is the same as the entropy of <math>X</math> and as the entropy of <math>Y</math>. (A very special case of this is when <math>X</math> and <math>Y</math> are the same random variable.)
直观地说，互信息衡量了<math>X</math> and <math>Y</math>的信息共享程度：它衡量了当已知其中一个变量后可以减少另一个变量多少的不确定性。例如，若<math>X</math>和<math>Y</math>是相互独立的，那么已知<math>X</math>不会得到关于<math>Y</math>的任何信息，反之亦然，因此它们之间的互信息为零。而另一种极端情况就是，若<math>X</math>是<math>Y</math>的'''<font color="#32CD32">确定函数</font>'''，而<math>X</math>也是<math>X</math>自身的确定函数，则<math>X</math>传递的所有信息都与<math>Y</math>共享：即已知<math>X</math>就可以知道<math>Y</math>的值，反之亦然。因此，在这种情况下，互信息与仅包含在<math>Y</math>（或<math>X</math>）中的不确定性相同，即<math>Y</math>（或<math>X</math>）的熵相同。此外，这种情况下互信息与<math>X</math>的熵和<math>Y</math>的熵相同。（一个非常特殊的情况是当<math>X</math>和<math>Y</math>是相同的随机变量。）
Using Jensen's inequality on the definition of mutual information we can show that <math>\operatorname{I}(X;Y)</math> is non-negative, i.e.
Using Jensen's inequality on the definition of mutual information we can show that <math>\operatorname{I}(X;Y)</math> is non-negative, i.e.
利用'''<font color="#ff8000">琴生不等式 Jensen's inequality</font>'''对互信息的定义进行推导，我们可以证明<math>\operatorname{I}(X;Y)</math>是非负的，即:
利用'''<font color="#ff8000">琴生不等式 Jensen's Inequality</font>'''对互信息的定义进行推导，我们可以证明<math>\operatorname{I}(X;Y)</math>是非负的，即:
<math>\operatorname{I}(X;Y) \ge 0</math>
<math>\operatorname{I}(X;Y) \ge 0</math>
其中<math>H(X)</math>和<math>H(Y)</math>是'''<font color="#ff8000">边际熵 Marginal entropy</font>'''，<math>H(X|Y)</math>和<math>H(Y|X)</math>表示'''<font color="#ff8000">条件熵 Conditional entropy</font>'''，<math>H(X,Y)</math>是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">联合熵 Joint entropy</font>'''。
其中<math>H(X)</math>和<math>H(Y)</math>是'''<font color="#ff8000">边际熵 Marginal Entropy</font>'''，<math>H(X|Y)</math>和<math>H(Y|X)</math>表示'''<font color="#ff8000">条件熵 Conditional Entropy</font>'''，<math>H(X,Y)</math>是<math>X</math>和<math>Y</math>的'''<font color="#ff8000">联合熵 Joint Entropy</font>'''。
mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, 𝑝𝑋⋅𝑝𝑌, from the joint distribution 𝑝(𝑋,𝑌), that is,
mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, 𝑝𝑋⋅𝑝𝑌, from the joint distribution 𝑝(𝑋,𝑌), that is,
互信息是边缘分布乘积的KL散度<math>D_{KL}</math>，也就是联合分布<math>p_{(X,Y)}</math>的乘积，即：
Note that here the Kullback–Leibler divergence involves integration over the values of the random variable <math>X</math> only, and the expression <math>D_\text{KL}(p_{X|Y} \parallel p_X)</math> still denotes a random variable because <math>Y</math> is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution <math>p_X</math> of <math>X</math> from the conditional distribution <math>p_{X|Y}</math> of <math>X</math> given <math>Y</math>: the more different the distributions <math>p_{X|Y}</math> and <math>p_X</math> are on average, the greater the information gain.
Note that here the Kullback–Leibler divergence involves integration over the values of the random variable <math>X</math> only, and the expression <math>D_\text{KL}(p_{X|Y} \parallel p_X)</math> still denotes a random variable because <math>Y</math> is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution <math>p_X</math> of <math>X</math> from the conditional distribution <math>p_{X|Y}</math> of <math>X</math> given <math>Y</math>: the more different the distributions <math>p_{X|Y}</math> and <math>p_X</math> are on average, the greater the information gain.
因此，互信息也可以理解为X的单变量分布<math>p_X</math>与给定<math>Y</math>的<math>X</math>的条件分布<math>p_{X|Y}</math>的相对熵的期望：平均分布<math>p_{X|Y}</math>和<math>p_X</math>的分布差异越大，信息增益越大。
因此，互信息也可以理解为X的单变量分布<math>p_X</math>与给定<math>Y</math>的<math>X</math>的条件分布<math>p_{X|Y}</math>的KL散度的期望：平均分布<math>p_{X|Y}</math>和<math>p_X</math>的分布差异越大，信息增益越大。
=== 互信息的贝叶斯估计 Bayesian estimation of mutual information ===
=== 互信息的贝叶斯估计 Bayesian estimation of mutual information ===
The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing 𝑝(𝑥,𝑦) to the fully factorized outer product 𝑝(𝑥)⋅𝑝(𝑦). In many problems, such as non-negative matrix factorization, one is interested in '''<font color="#32CD32">less extreme factorizations</font>'''; specifically, one wishes to compare 𝑝(𝑥,𝑦) to a low-rank matrix approximation in some unknown variable 𝑤; that is, to what degree one might have
The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing 𝑝(𝑥,𝑦) to the fully factorized outer product 𝑝(𝑥)⋅𝑝(𝑦). In many problems, such as non-negative matrix factorization, one is interested in '''<font color="#32CD32">less extreme factorizations</font>'''; specifically, one wishes to compare 𝑝(𝑥,𝑦) to a low-rank matrix approximation in some unknown variable 𝑤; that is, to what degree one might have
互信息的KL散度公式是基于这样一个结论的：人们会更关注将<math>p(x,y)</math>与完全分解的'''<font color="#ff8000">外积 Outer Product</font>'''<math>p(x) \cdot p(y)</math>进行比较。在许多问题中，例如'''<font color="#ff8000">非负矩阵因式分解 Non-negative matrix factorization</font>'''中，人们对'''<font color="#32CD32">较不极端的</font>'''因式分解感兴趣；具体地说，人们希望将<math>p(x,y)</math>与某个未知变量<math>w</math>中的低秩矩阵近似进行比较；也就是说，在多大程度上可能会有这样的结果：
:<math>p(x,y)\approx \sum_w p^\prime (x,w) p^{\prime\prime}(w,y)</math>
:<math>p(x,y)\approx \sum_w p^\prime (x,w) p^{\prime\prime}(w,y)</math>
Alternately, one might be interested in knowing how much more information 𝑝(𝑥,𝑦) carries over its factorization. In such a case, the excess information that the full distribution 𝑝(𝑥,𝑦) carries over the matrix factorization is given by the Kullback-Leibler divergence
Alternately, one might be interested in knowing how much more information 𝑝(𝑥,𝑦) carries over its factorization. In such a case, the excess information that the full distribution 𝑝(𝑥,𝑦) carries over the matrix factorization is given by the Kullback-Leibler divergence
另一方面，人们可能有兴趣知道在因子分解过程中，有<math>p(x,y)</math>携带了多少信息。在这种情况下，全分布<math>p(x,y)</math>通过矩阵因子分解所携带的多余信息由Kullback-Leibler散度给出
另一方面，人们可能有兴趣知道在因子分解过程中，有<math>p(x,y)</math>携带了多少信息。在这种情况下，全分布<math>p(x,y)</math>通过矩阵因子分解所携带的多余信息由KL散度给出
:<math>\operatorname{I}_{LRMA} = \sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}}
:<math>\operatorname{I}_{LRMA} = \sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}}
In a set-theoretic interpretation of information (see the figure for Conditional entropy), this is effectively the Jaccard distance between 𝑋 and 𝑌.
In a set-theoretic interpretation of information (see the figure for Conditional entropy), this is effectively the Jaccard distance between 𝑋 and 𝑌.
在信息的集合论解释中（参见条件熵的图），这实际上就是<math>X</math>和<math>Y</math>之间的'''<font color="#ff8000">杰卡德距离 Jaccard distance</font>'''。
在信息的集合论解释中（参见条件熵的图），这实际上就是<math>X</math>和<math>Y</math>之间的'''<font color="#ff8000">杰卡德距离 Jaccard Distance</font>'''。
Several generalizations of mutual information to more than two random variables have been proposed, such as total correlation (or multi-information) and interaction information. The expression and study of multivariate higher-degree mutual-information was achieved in two seemingly independent works: McGill (1954) who called these functions “interaction information”, and Hu Kuo Ting (1962) who also first proved the possible negativity of mutual-information for degrees higher than 2 and justified algebraically the intuitive correspondence to Venn diagrams
Several generalizations of mutual information to more than two random variables have been proposed, such as total correlation (or multi-information) and interaction information. The expression and study of multivariate higher-degree mutual-information was achieved in two seemingly independent works: McGill (1954) who called these functions “interaction information”, and Hu Kuo Ting (1962) who also first proved the possible negativity of mutual-information for degrees higher than 2 and justified algebraically the intuitive correspondence to Venn diagrams
目前提出了许多将互信息推广到两个以上随机变量的方法，如'''<font color="#ff8000">全相关 Total correlation</font>'''（或'''<font color="#ff8000">多信息 Multi-information</font>'''）以及'''<font color="#ff8000">交互信息 Interaction information</font>'''。多元高阶互信息的表达和研究是在两部看似独立的著作中实现的：McGill（1954年）在文献[8]中将这些函数统称为“互信息”，胡国亭（1962年）也在文献[9]中首次证明了大于2度的互信息可能是负的，并在文献[10]中用代数的方法证明了互信息和维恩图的直观对应关系。
目前提出了许多将互信息推广到两个以上随机变量的方法，如'''<font color="#ff8000">全相关 Total Correlation</font>'''（或'''<font color="#ff8000">多信息 Multi-Information</font>'''）以及'''<font color="#ff8000">交互信息 Interaction Information</font>'''。多元高阶互信息的表达和研究是在两部看似独立的著作中实现的：McGill（1954年）在文献[8]中将这些函数统称为“互信息”，胡国亭（1962年）也在文献[9]中首次证明了大于2度的互信息可能是负的，并在文献[10]中用代数的方法证明了互信息和维恩图的直观对应关系。
For 3 variables, Brenner et al. applied multivariate mutual information to neural coding and called its negativity "synergy" and Watkinson et al. applied it to genetic expression . For arbitrary k variables, Tapia et al. applied multivariate mutual information to gene expression . '''<font color="#32CD32">The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints </font>'''.
For 3 variables, Brenner et al. applied multivariate mutual information to neural coding and called its negativity "synergy" and Watkinson et al. applied it to genetic expression . For arbitrary k variables, Tapia et al. applied multivariate mutual information to gene expression . '''<font color="#32CD32">The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints </font>'''.
对于3个变量，Brenner等人将多变量互信息应用到神经编码中，并将其称为'''<font color="#ff8000">负面“协同作用” Negativity "synergy"</font>'''，接着Watkinson 等人将其应用到基因表达上。对于任意k个变量，Tapia 等人将多元互信息应用于基因表达——'''<font color="#32CD32">正性对应于一般化成对相关性的关系，无效性对应于一个精确的独立性概念，负性检测高维“涌现”关系和聚合数据点</font>'''。
对于3个变量，Brenner等人将多变量互信息应用到神经编码中，并将其称为'''<font color="#ff8000">负面“协同作用” Negativity "Synergy"</font>'''，接着Watkinson 等人将其应用到基因表达上。对于任意k个变量，Tapia 等人将多元互信息应用于基因表达——'''<font color="#32CD32">正性对应于一般化成对相关性的关系，无效性对应于一个精确的独立性概念，负性检测高维“涌现”关系和聚合数据点</font>'''。
One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in feature selection.
One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in feature selection.
目前已经提出了一种能够最大化联合分布与其他目标变量之间的互信息的高维推广方案，该方法可用于'''<font color="#ff8000"> 特征选择 Feature selection</font>'''。
目前已经提出了一种能够最大化联合分布与其他目标变量之间的互信息的高维推广方案，该方法可用于'''<font color="#ff8000"> 特征选择 Feature Selection</font>'''。
Mutual information is also used in the area of signal processing as a [[Similarity measure|measure of similarity]] between two signals. For example, FMI metric<ref>{{cite journal | last1 = Haghighat | first1 = M. B. A. | last2 = Aghagolzadeh | first2 = A. | last3 = Seyedarabi | first3 = H. | year = 2011 | title = A non-reference image fusion metric based on mutual information of image features | doi = 10.1016/j.compeleceng.2011.07.012 | journal = Computers & Electrical Engineering | volume = 37 | issue = 5| pages = 744–756 }}</ref> is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The [[Matlab]] code for this metric can be found at.<ref>{{cite web|url=http://www.mathworks.com/matlabcentral/fileexchange/45926-feature-mutual-information-fmi-image-fusion-metric|title=Feature Mutual Information (FMI) metric for non-reference image fusion - File Exchange - MATLAB Central|author=|date=|website=www.mathworks.com|accessdate=4 April 2018}}</ref>
Mutual information is also used in the area of signal processing as a [[Similarity measure|measure of similarity]] between two signals. For example, FMI metric<ref>{{cite journal | last1 = Haghighat | first1 = M. B. A. | last2 = Aghagolzadeh | first2 = A. | last3 = Seyedarabi | first3 = H. | year = 2011 | title = A non-reference image fusion metric based on mutual information of image features | doi = 10.1016/j.compeleceng.2011.07.012 | journal = Computers & Electrical Engineering | volume = 37 | issue = 5| pages = 744–756 }}</ref> is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The [[Matlab]] code for this metric can be found at.<ref>{{cite web|url=http://www.mathworks.com/matlabcentral/fileexchange/45926-feature-mutual-information-fmi-image-fusion-metric|title=Feature Mutual Information (FMI) metric for non-reference image fusion - File Exchange - MATLAB Central|author=|date=|website=www.mathworks.com|accessdate=4 April 2018}}</ref>
Mutual information is also used in the area of signal processing as a measure of similarity between two signals. For example, FMI metric is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The Matlab code for this metric can be found at.
Mutual information is also used in the area of signal processing as a measure of similarity between two signals. For example, FMI metric is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The Matlab code for this metric can be found at.
互信息也用于信号处理领域，用来进行两个信号之间的'''<font color="#ff8000">相似性度量 Similarity measure</font>'''。例如，FMI 度量是一种利用互信息来度量融合图像包含的关于源图像的信息量的图像融合性能度量。这个度量的 Matlab 代码可以在参考文献[17]中找到。
互信息也用于信号处理领域，用来进行两个信号之间的'''<font color="#ff8000">相似性度量 Similarity Measure</font>'''。例如，FMI 度量是一种利用互信息来度量融合图像包含的关于源图像的信息量的图像融合性能度量。这个度量的 Matlab 代码可以在参考文献[17]中找到。
=== 定向信息 Directed information ===
=== 定向信息 Directed information ===
Directed information, I(𝑋𝑛→𝑌𝑛), measures the amount of information that flows from the process 𝑋𝑛 to 𝑌𝑛, where 𝑋𝑛 denotes the vector 𝑋1,𝑋2,...,𝑋𝑛 and 𝑌𝑛 denotes 𝑌1,𝑌2,...,𝑌𝑛. The term directed information was coined by James Massey and is defined as：
Directed information, I(𝑋𝑛→𝑌𝑛), measures the amount of information that flows from the process 𝑋𝑛 to 𝑌𝑛, where 𝑋𝑛 denotes the vector 𝑋1,𝑋2,...,𝑋𝑛 and 𝑌𝑛 denotes 𝑌1,𝑌2,...,𝑌𝑛. The term directed information was coined by James Massey and is defined as：
'''<font color="#ff8000">定向信息 Directed information</font>'''<math>\operatorname{I}\left(X^n \to Y^n\right)</math>度量从<math>X^n</math>流向<math>Y^n</math>的过程中的信息量，其中<math>X^n</math>表示为向量<math>X_1, X_2, ..., X_n</math>，<math>Y^n</math>表示为<math>Y_1, Y_2, ..., Y_n</math>。定向信息这个术语是由 James Massey 创造的，它被定义为：
'''<font color="#ff8000">定向信息 Directed Information</font>'''<math>\operatorname{I}\left(X^n \to Y^n\right)</math>度量从<math>X^n</math>流向<math>Y^n</math>的过程中的信息量，其中<math>X^n</math>表示为向量<math>X_1, X_2, ..., X_n</math>，<math>Y^n</math>表示为<math>Y_1, Y_2, ..., Y_n</math>。定向信息这个术语是由 James Massey 创造的，它被定义为：
Note that if 𝑛=1, the directed information becomes the mutual information. Directed information has many applications in problems where causality plays an important role, such as capacity of channel with feedback.
Note that if 𝑛=1, the directed information becomes the mutual information. Directed information has many applications in problems where causality plays an important role, such as capacity of channel with feedback.
注意，当<math>n=1</math>时，则定向信息成为互信息。定向信息在因果关系问题中有着广泛的应用，如反馈'''<font color="#ff8000">信道容量问题 Channel capacity</font>'''。
注意，当<math>n=1</math>时，则定向信息成为互信息。定向信息在因果关系问题中有着广泛的应用，如反馈'''<font color="#ff8000">信道容量问题 Channel Capacity</font>'''。
=== 标准化变形 Normalized variants ===
=== 标准化变形 Normalized variants ===
when one variable becomes completely redundant with the knowledge of the other. See also Redundancy (information theory).
when one variable becomes completely redundant with the knowledge of the other. See also Redundancy (information theory).
当一个变量与另一个变量的知识完全多余时。参见'''<font color="#ff8000">冗余 Redundancy</font>'''（信息论）。
Another symmetrical measure is the symmetric uncertainty , given by
Another symmetrical measure is the symmetric uncertainty , given by
另一个对称度量是''对称不确定度''，由下式表示：
which represents the harmonic mean of the two uncertainty coefficients <math>C_{XY}, C_{YX}</math>.
which represents the harmonic mean of the two uncertainty coefficients <math>C_{XY}, C_{YX}</math>.
它表示两个不确定系数的'''<font color="#ff8000">调和平均数 Harmonic Mean</font>'''。
If we consider mutual information as a special case of the total correlation or dual total correlation, the normalized version are respectively,
If we consider mutual information as a special case of the total correlation or dual total correlation, the normalized version are respectively,
如果我们把互信息看作是'''<font color="#ff8000">总相关 Total correlation</font>'''或'''<font color="#ff8000">对偶总相关 Dual total correlation</font>'''的特殊情况，则其标准化版本分别为,
如果我们把互信息看作是'''<font color="#ff8000">总相关 Total Correlation</font>'''或'''<font color="#ff8000">对偶总相关 Dual Total Correlation</font>'''的特殊情况，则其标准化版本分别为,
:<math>\frac{\operatorname{I}(X;Y)}{\min\left[ H(X),H(Y)\right]}</math> and <math>\frac{\operatorname{I}(X;Y)}{H(X,Y)} \; .</math>
:<math>\frac{\operatorname{I}(X;Y)}{\min\left[ H(X),H(Y)\right]}</math> and <math>\frac{\operatorname{I}(X;Y)}{H(X,Y)} \; .</math>
This normalized version also known as Information Quality Ratio (IQR) which quantifies the amount of information of a variable based on another variable against total uncertainty:
This normalized version also known as Information Quality Ratio (IQR) which quantifies the amount of information of a variable based on another variable against total uncertainty:
这个标准化版本也被称为'''<font color="#ff8000">信息质量比率 Information Quality Ratio，IQR</font>''' ，它根据另一个变量量化了一个变量的信息量，来对抗总的不确定性:
这个标准化版本也被称为'''<font color="#ff8000">信息质量比率 Information Quality Ratio, IQR</font>''' ，它根据另一个变量量化了一个变量的信息量，来对抗总的不确定性:
:<math>IQR(X, Y) = \operatorname{E}[\operatorname{I}(X;Y)]
:<math>IQR(X, Y) = \operatorname{E}[\operatorname{I}(X;Y)]
There's a normalization which derives from first thinking of mutual information as an analogue to [[covariance]] (thus [[Entropy (information theory)|Shannon entropy]] is analogous to [[variance]]). Then the normalized mutual information is calculated akin to the [[Pearson product-moment correlation coefficient|Pearson correlation coefficient]],
There's a normalization which derives from first thinking of mutual information as an analogue to [[covariance]] (thus [[Entropy (information theory)|Shannon entropy]] is analogous to [[variance]]). Then the normalized mutual information is calculated akin to the [[Pearson product-moment correlation coefficient|Pearson correlation coefficient]],
有一个标准化的名字——它起源于最初把互信息看作是'''<font color="#ff8000">协方差 Covariance</font>'''的类比（因此香农熵类似于方差）。然后计算归一化互信息类似于'''<font color="#ff8000">皮尔森相关系数 Pearson product-moment</font>'''：
有一个标准化的名字——它起源于最初把互信息看作是'''<font color="#ff8000">协方差 Covariance</font>'''的类比（因此香农熵类似于方差）。然后计算归一化互信息类似于'''<font color="#ff8000">皮尔森相关系数 Pearson Product-moment</font>'''：
A probability distribution can be viewed as a partition of a set. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? The adjusted mutual information or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to the adjusted Rand index of two different partitions of a set.
A probability distribution can be viewed as a partition of a set. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? The adjusted mutual information or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to the adjusted Rand index of two different partitions of a set.
概率分布可以被看作是集合划分。然后有人可能会问: 如果一个集合被随机分割，概率的分布会是什么？相互信息的期望值是什么？我们用'''<font color="#ff8000">调整后的互信息 Adjusted mutual information</font>'''或 AMI 减去 MI 的期望值，这样当两个不同的分布是随机的时候 AMI 为零，当两个分布是相同的时候 AMI 为零。AMI的定义类似于一个集合的两个不同分区的'''<font color="#ff8000">调整后的Rand指数 Adjusted Rand index</font>'''。
概率分布可以被看作是集合划分。然后有人可能会问: 如果一个集合被随机分割，概率的分布会是什么？相互信息的期望值是什么？我们用'''<font color="#ff8000">调整后的互信息 Adjusted Mutual Information</font>'''或 AMI 减去 MI 的期望值，这样当两个不同的分布是随机的时候 AMI 为零，当两个分布是相同的时候 AMI 为零。AMI的定义类似于一个集合的两个不同分区的'''<font color="#ff8000">调整后的Rand指数 Adjusted Rand Index</font>'''。
=== 绝对互信息 Absolute mutual information ===<!-- This section is linked from Kolmogorov complexity -->
=== 绝对互信息 Absolute mutual information ===<!-- This section is linked from Kolmogorov complexity -->
Using the ideas of Kolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:
Using the ideas of Kolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:
利用'''<font color="#ff8000">柯氏复杂性 Kolmogorov complexity</font>'''的思想，我们可以考虑两个序列的互信息，这两个序列独立于任何概率分布序列:
利用'''<font color="#ff8000">柯氏复杂性 Kolmogorov Complexity</font>'''的思想，我们可以考虑两个序列的互信息，这两个序列独立于任何概率分布序列:
To establish that this quantity is symmetric up to a logarithmic factor (I𝐾(𝑋;𝑌)≈I𝐾(𝑌;𝑋)) one requires the chain rule for Kolmogorov complexity.Approximations of this quantity via compression can be used to define a distance measure to perform a hierarchical clustering of sequences without having any domain knowledge of the sequences.
To establish that this quantity is symmetric up to a logarithmic factor (I𝐾(𝑋;𝑌)≈I𝐾(𝑌;𝑋)) one requires the chain rule for Kolmogorov complexity.Approximations of this quantity via compression can be used to define a distance measure to perform a hierarchical clustering of sequences without having any domain knowledge of the sequences.
为了确定这个量在对数因子<math>\operatorname{I}_K(X;Y) \approx \operatorname{I}_K(Y;X)</math>是对称的，需要'''<font color="#ff8000"> Kolmogorov复杂性的链式规则 Chain rule for Kolmogorov complexity</font>'''。通过压缩对这个量的近似值可以用来定义'''<font color="#ff8000">距离度量 Distance measure</font>'''来执行序列的'''<font color="#ff8000">层次聚类 Hierarchical clustering</font>'''，而不需要序列的任何领域知识。
为了确定这个量在对数因子<math>\operatorname{I}_K(X;Y) \approx \operatorname{I}_K(Y;X)</math>是对称的，需要'''<font color="#ff8000"> 柯氏复杂性的链式规则 Chain Rule for Kolmogorov Complexity</font>'''。通过压缩对这个量的近似值可以用来定义'''<font color="#ff8000">距离度量 Distance Measure</font>'''来执行序列的'''<font color="#ff8000">层次聚类 Hierarchical Clustering</font>'''，而不需要序列的任何领域知识。
=== 线性相关 Linear correlation ===
=== 线性相关 Linear correlation ===
Unlike correlation coefficients, such as the product moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for <math>X</math> and <math>Y</math> is a bivariate normal distribution (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between <math>\operatorname{I}</math> and the correlation coefficient <math>\rho</math> .
Unlike correlation coefficients, such as the product moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for <math>X</math> and <math>Y</math> is a bivariate normal distribution (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between <math>\operatorname{I}</math> and the correlation coefficient <math>\rho</math> .
互信息不同于相关系数，如'''<font color="#ff8000">积矩相关系数 Product Moment Correlation Coefficient</font>'''，互信息包含所有相关信息ーー线性和非线性ーー而不仅仅是相关系数的线性相关。然而，在<math>X</math>和<math>Y</math>的联合分布是'''<font color="#ff8000">二元正态分布 Bivariate Normal Distribution</font>'''（特别是边际分布都是正态分布）的狭义情况下，<math>\operatorname{I}</math>与相关系数<math>\rho</math>之间存在精确的关系。
:<math>\operatorname{I} = -\frac{1}{2} \log\left(1 - \rho^2\right)</math>
:<math>\operatorname{I} = -\frac{1}{2} \log\left(1 - \rho^2\right)</math>
When 𝑋 and 𝑌 are limited to be in a discrete number of states, observation data is summarized in a contingency table, with row variable 𝑋 (or 𝑖) and column variable 𝑌 (or 𝑗). Mutual information is one of the measures of association or correlation between the row and column variables. Other measures of association include Pearson's chi-squared test statistics, G-test statistics, etc. In fact, mutual information is equal to G-test statistics divided by 2𝑁, where 𝑁 is the sample size.
When 𝑋 and 𝑌 are limited to be in a discrete number of states, observation data is summarized in a contingency table, with row variable 𝑋 (or 𝑖) and column variable 𝑌 (or 𝑗). Mutual information is one of the measures of association or correlation between the row and column variables. Other measures of association include Pearson's chi-squared test statistics, G-test statistics, etc. In fact, mutual information is equal to G-test statistics divided by 2𝑁, where 𝑁 is the sample size.
当<math>X</math>和<math>Y</math>被限制为离散状态时，观测数据汇总在'''<font color="#ff8000">列联表 Contingency table</font>'''中，列变量为行变量<math>X</math>（或<math>i</math>）和列变量<math>Y</math>（或<math>j</math>）。互信息是行和列变量之间关联或相关性的度量之一。其他关联度量包括Pearson卡方检验统计量、'''<font color="#ff8000">G检验 G-test</font>'''统计量等。事实上，互信息等于G检验统计量除以<math>2N</math>，其中<math>N</math>为样本量。
当<math>X</math>和<math>Y</math>被限制为离散状态时，观测数据汇总在'''<font color="#ff8000">列联表 Contingency Table</font>'''中，列变量为行变量<math>X</math>（或<math>i</math>）和列变量<math>Y</math>（或<math>j</math>）。互信息是行和列变量之间关联或相关性的度量之一。其他关联度量包括Pearson卡方检验统计量、'''<font color="#ff8000">G检验 G-Test</font>'''统计量等。事实上，互信息等于G检验统计量除以<math>2N</math>，其中<math>N</math>为样本量。
== 应用 Applications ==
== 应用 Applications ==
where <math>f_{XY}</math> is the number of times the bigram xy appears in the corpus, <math>f_{X}</math> is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.
where <math>f_{XY}</math> is the number of times the bigram xy appears in the corpus, <math>f_{X}</math> is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.
其中<math>f_{XY}</math>是 bigram XY 在语料库中出现的次数，<math>f_{X}</math>是 unigram x 在语料库中出现的次数，b 是 bigrams 的总数，u 是 unigrams 的总数。
其中<math>f_{XY}</math>是 二元语法 XY 在语料库中出现的次数，<math>f_{X}</math>是一元模型x在语料库中出现的次数，b 是二元语法的总数，u 是一元模型的总数。
* In [[telecommunications]], the [[channel capacity]] is equal to the mutual information, maximized over all input distributions.
* In [[telecommunications]], the [[channel capacity]] is equal to the mutual information, maximized over all input distributions.
Discriminative training procedures for hidden Markov models have been proposed based on the maximum mutual information (MMI) criterion.
Discriminative training procedures for hidden Markov models have been proposed based on the maximum mutual information (MMI) criterion.
现在已经提出了基于最大互信息（MMI）准则的'''<font color="#ff8000">隐马尔可夫模型 Hidden Markov Model</font>'''判别训练方法。
* [[Nucleic acid secondary structure|RNA secondary structure]] prediction from a [[multiple sequence alignment]].
* [[Nucleic acid secondary structure|RNA secondary structure]] prediction from a [[multiple sequence alignment]].
Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.
Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.
互信息用于确定数据集中两个不同'''<font color="#ff8000">聚类 Clusterings</font>'''的相似性。因此，它与传统的Rand指数相比具有一定的优势。
* Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within <math>N</math> words of another, goes up with <math>N</math>.
* Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within <math>N</math> words of another, goes up with <math>N</math>.
Mutual information of words is often used as a significance function for the computation of collocations in corpus linguistics. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within 𝑁 words of another, goes up with 𝑁.
Mutual information of words is often used as a significance function for the computation of collocations in corpus linguistics. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within 𝑁 words of another, goes up with 𝑁.
在'''<font color="#ff8000">语料库语言学 Corpus Linguistics</font>'''中，词的互信息常常被用作计算搭配的意义函数。这增加了复杂性，即没有一个单词实例是两个不同单词的实例；相反，我们统计两个单词相邻或非常接近的实例；这稍微使计算复杂化，因为一个单词出现在另一个单词的<math>N</math>单词中的预期概率会增加。
* Mutual information is used in [[medical imaging]] for [[image registration]]. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same [[coordinate system]] as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
* Mutual information is used in [[medical imaging]] for [[image registration]]. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same [[coordinate system]] as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
Mutual information is used in medical imaging for image registration. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same coordinate system as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
Mutual information is used in medical imaging for image registration. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same coordinate system as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
在'''<font color="#ff8000">医学图像 medical imaging</font>'''中，利用互信息进行'''<font color="#ff8000">图像配准 Image Registration</font>'''。给定一个参考图像（例如，脑部扫描），以及需要将第二个图像放入与参考图像相同的'''<font color="#ff8000">坐标系 Coordinate System</font>'''中，该图像会发生变形，直到其与参考图像之间的互信息最大化。
* Detection of [[phase synchronization]] in [[time series]] analysis
* Detection of [[phase synchronization]] in [[time series]] analysis
Detection of phase synchronization in time series analysis
Detection of phase synchronization in time series analysis
时间序列分析中的'''<font color="#ff8000">相位同步 Phase Synchronization</font>'''检测。
* In the [[infomax]] method for neural-net and other machine learning, including the infomax-based [[Independent component analysis]] algorithm
* In the [[infomax]] method for neural-net and other machine learning, including the infomax-based [[Independent component analysis]] algorithm
In the infomax method for neural-net and other machine learning, including the infomax-based Independent component analysis algorithm.
In the infomax method for neural-net and other machine learning, including the infomax-based Independent component analysis algorithm.
在'''<font color="#ff8000">信息极大化 Infomax</font>'''方法中用于神经网络等机器学习，包括基于信息极大化的'''<font color="#ff8000">独立成分分析 Independent Component Analysis</font>'''算法
* Average mutual information in [[delay embedding theorem]] is used for determining the ''embedding delay'' parameter.
* Average mutual information in [[delay embedding theorem]] is used for determining the ''embedding delay'' parameter.
Average mutual information in delay embedding theorem is used for determining the embedding delay parameter.
Average mutual information in delay embedding theorem is used for determining the embedding delay parameter.
利用'''<font color="#ff8000">延迟嵌入定理 Delay Embedding Theorem</font>'''中的平均互信息确定嵌入延迟参数。
* Mutual information between [[genes]] in [[microarray|expression microarray]] data is used by the ARACNE algorithm for reconstruction of [[gene regulatory network|gene networks]].
* Mutual information between [[genes]] in [[microarray|expression microarray]] data is used by the ARACNE algorithm for reconstruction of [[gene regulatory network|gene networks]].
Mutual information between genes in expression microarray data is used by the ARACNE algorithm for reconstruction of gene networks.
Mutual information between genes in expression microarray data is used by the ARACNE algorithm for reconstruction of gene networks.
ARACNE算法利用表达微阵列数据中基因间的互信息来重构'''<font color="#ff8000">基因网络 Gene Networks</font>'''。
In statistical mechanics, Loschmidt's paradox may be expressed in terms of mutual information.[27][28] Loschmidt noted that it must be impossible to determine a physical law which lacks time reversal symmetry (e.g. the second law of thermodynamics) only from physical laws which have this symmetry. He pointed out that the H-theorem of Boltzmann made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in phase space, then Liouville's theorem implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by Boltzmann's constant).
In statistical mechanics, Loschmidt's paradox may be expressed in terms of mutual information.[27][28] Loschmidt noted that it must be impossible to determine a physical law which lacks time reversal symmetry (e.g. the second law of thermodynamics) only from physical laws which have this symmetry. He pointed out that the H-theorem of Boltzmann made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in phase space, then Liouville's theorem implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by Boltzmann's constant).
在统计力学中，'''<font color="#ff8000">洛施密特悖论 Loschmidt's Paradox</font>'''可以用互信息来表示。洛施密特指出，只有从具有这种对称性的物理定律中确定缺乏时间反转对称性的物理定律（例如'''<font color="#ff8000">热力学第二定律 Second Law of Thermodynamics</font>'''）是不可能的。他指出，玻尔兹曼的H-定理假设气体中粒子的速度是永久不相关的，这就消除了H-定理固有的时间对称性。可以证明，如果系统在相空间中用概率密度来描述，那么'''<font color="#ff8000">刘维尔定理 Liouville's Theorem</font>'''意味着分布的联合信息（联合熵的负）在时间上保持不变。关节信息等于互信息加上每个粒子坐标的所有边缘信息（负的边缘熵）之和。玻尔兹曼的假设相当于在熵的计算中忽略了相互信息，从而得到了热力学熵（除以玻尔兹曼常数）。
The mutual information is used to learn the structure of Bayesian networks/dynamic Bayesian networks, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:[29] learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.
The mutual information is used to learn the structure of Bayesian networks/dynamic Bayesian networks, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:[29] learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.
互信息用于学习'''<font color="#ff8000">贝叶斯网络 Bayesian Network</font>'''/'''<font color="#ff8000">动态贝叶斯网络 Dynamic Bayesian Network</font>'''的结构，被认为是用来解释随机变量之间的因果关系，如GlobalMIT工具包[29]用互信息检验准则学习全局最优动态贝叶斯网络。
Popular cost function in decision tree learning.
Popular cost function in decision tree learning.
作为'''<font color="#ff8000">决策树学习 Decision Tree Learning</font>'''中常用的代价函数。
* The mutual information is used in [[cosmology]] to test the influence of large-scale environments on galaxy properties in the [[Galaxy Zoo]].
* The mutual information is used in [[cosmology]] to test the influence of large-scale environments on galaxy properties in the [[Galaxy Zoo]].
The mutual information is used in cosmology to test the influence of large-scale environments on galaxy properties in the Galaxy Zoo.
The mutual information is used in cosmology to test the influence of large-scale environments on galaxy properties in the Galaxy Zoo.
在'''<font color="#32CD32">星系 Galaxy Zoo</font>'''中，利用互信息在宇宙学中测试大尺度环境对星系性质的影响。
在'''<font color="#32CD32">星系 Galaxy Zoo</font>'''中，利用互信息在'''<font color="#ff8000">宇宙学 Cosmology</font>'''中测试大尺度环境对星系性质的影响。
The mutual information was used in Solar Physics to derive the solar differential rotation profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements.
The mutual information was used in Solar Physics to derive the solar differential rotation profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements.
在'''<font color="#ff8000">太阳物理学 Solar Physics</font>'''中，相互信息被用于推导太阳差分自转剖面图、太阳黑子的旅行时间偏差图和安静太阳测量的时间-距离图。
* Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.<ref name=iic>[https://arxiv.org/abs/1807.06653 Invariant Information Clustering for Unsupervised Image Classification and Segmentation] by Xu Ji, Joao Henriques and Andrea Vedaldi</ref>
* Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.<ref name=iic>[https://arxiv.org/abs/1807.06653 Invariant Information Clustering for Unsupervised Image Classification and Segmentation] by Xu Ji, Joao Henriques and Andrea Vedaldi</ref>
== 参见 See also ==
== 参见 See also ==
* [[Pointwise mutual information]]
* [[Pointwise mutual information 点态互信息]]
* [[Quantum mutual information 量子互信息]]
* [[Quantum mutual information]]
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）记得翻译一下这里
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）记得翻译一下这里
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）记得图片以及专业名词的标准格式~以及see also的完整
--[[用户:趣木木|趣木木]]（[[用户讨论:趣木木|讨论]]）记得图片以及专业名词的标准格式~以及see also的完整
* {{cite journal
* {{cite journal
| last1 = Wells
| last1 = Wells
| first1 = W.M. III
| first1 = W.M. III
| last2 = Viola
| last2 = Viola
| first2 = P.
| first2 = P.
| last3 = Atsumi
| last3 = Atsumi
| first3 = H.
| first3 = H.
| last4 = Nakajima
| last4 = Nakajima
| first4 = S.
| first4 = S.
| last5 = Kikinis
| last5 = Kikinis
| first5 = R.
| first5 = R.
| title = Multi-modal volume registration by maximization of mutual information
| title = Multi-modal volume registration by maximization of mutual information
| journal = Medical Image Analysis
| journal = Medical Image Analysis
| volume = 1
| volume = 1
| issue = 1
| issue = 1
| pages = 35–51
| pages = 35–51
| year = 1996
| year = 1996
| pmid = 9873920
| pmid = 9873920
| doi = 10.1016/S1361-8415(01)80004-9
| doi = 10.1016/S1361-8415(01)80004-9
| url = http://www.ai.mit.edu/people/sw/papers/mia.pdf
| url = http://www.ai.mit.edu/people/sw/papers/mia.pdf
| ref = harv
| ref = harv
| access-date = 2010-08-05
| access-date = 2010-08-05
| archive-url = https://web.archive.org/web/20080906201633/http://www.ai.mit.edu/people/sw/papers/mia.pdf
| archive-url = https://web.archive.org/web/20080906201633/http://www.ai.mit.edu/people/sw/papers/mia.pdf
| archive-date = 2008-09-06
| archive-date = 2008-09-06
| url-status = dead
| url-status = dead
}}
}}