互信息 - 版本历史

2024年5月7日 (二) 03:30 思无涯咿呀咿呀

2024-05-07T03:30:57Z

Vicky：/* 应用 Applications */

2021-02-18T01:49:02Z

应用 Applications

Vicky：/* 应用 Applications */

2021-02-18T01:48:32Z

应用 Applications

Vicky：/* 变形 Variations */

2021-02-18T01:43:03Z

变形 Variations

Vicky：/* 独立性假设 Independence assumptions */

2021-02-07T11:36:14Z

独立性假设 Independence assumptions

Vicky：/* 互信息的贝叶斯估计 Bayesian estimation of mutual information */

2021-02-07T09:38:49Z

互信息的贝叶斯估计 Bayesian estimation of mutual information

Vicky：/* 互信息的贝叶斯估计 Bayesian estimation of mutual information */

2021-02-07T09:11:02Z

互信息的贝叶斯估计 Bayesian estimation of mutual information

Vicky：/* 互信息的贝叶斯估计 Bayesian estimation of mutual information */

2021-02-07T08:48:03Z

互信息的贝叶斯估计 Bayesian estimation of mutual information

Vicky：/* 定义 Definition */

2021-02-07T08:40:53Z

定义 Definition

2021年2月6日 (六) 14:43 Flipped

2021-02-06T14:43:28Z

显示更改

←上一版本		2024年5月7日 (二) 03:30的版本
第1,302行：		第1,302行：
	<small>This page was moved from [[wikipedia:en:Mutual information]]. Its edit history can be viewed at [[互信息/edithistory]]</small></noinclude>		<small>This page was moved from [[wikipedia:en:Mutual information]]. Its edit history can be viewed at [[互信息/edithistory]]</small></noinclude>

−	[[Category:~~待整理页面~~]]	+	[[Category:待润色]]

@@ 第1,058行： / 第1,058行： @@
 Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.
-互信息用于确定数据集中两个不同聚类的相似性。因此，它与传统的'''<font color="#ff8000">兰德指数 Rand index</font>'''相比具有一定的优势。
+互信息用于确定数据集中两个不同聚类的相似性。因此，它与传统的Rand指数相比具有一定的优势。
 * Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within <math>N</math> words of another, goes up with <math>N</math>.

@@ 第1,058行： / 第1,058行： @@
 Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.
-互信息用于确定数据集中两个不同聚类的相似性。因此，它与传统的Rand指数相比具有一定的优势。
+互信息用于确定数据集中两个不同聚类的相似性。因此，它与传统的'''<font color="#ff8000">兰德指数 Rand index</font>'''相比具有一定的优势。
 * Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within <math>N</math> words of another, goes up with <math>N</math>.

@@ 第350行： / 第350行： @@
 在过程<math> W </math>中，<math>w</math>只有一个值的极端情况下，可以使用传统的互信息定义。
-== 变形 Variations ==
+== 变种 Variations ==
 Several variations on mutual information have been proposed to suit various needs.  Among these are normalized variants and generalizations to more than two variables.
@@ 第356行： / 第356行： @@
 Several variations on mutual information have been proposed to suit various needs.  Among these are normalized variants and generalizations to more than two variables.
-为了适应不同的需要，已经提出了几种互信息的变形。其中包括变量归一化和对两个以上变量的泛化。
+为了适应不同的需要，已经提出了几种互信息的变种。其中包括变量归一化和对两个以上变量的泛化。

@@ 第322行： / 第322行： @@
 The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing <math>p(x,y)</math> to the fully factorized [[outer product]] <math>p(x) \cdot p(y)</math>. In many problems, such as [[non-negative matrix factorization]], one is interested in less extreme factorizations; specifically, one wishes to compare <math>p(x,y)</math> to a low-rank matrix approximation in some unknown variable <math>w</math>; that is, to what degree one might have
-The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing 𝑝(𝑥,𝑦) to the fully factorized outer product 𝑝(𝑥)⋅𝑝(𝑦). In many problems, such as non-negative matrix factorization, one is interested in '''<font color="#32CD32">less extreme factorizations</font>'''; specifically, one wishes to compare 𝑝(𝑥,𝑦) to a low-rank matrix approximation in some unknown variable 𝑤; that is, to what degree one might have
+The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing 𝑝(𝑥,𝑦) to the fully factorized outer product 𝑝(𝑥)⋅𝑝(𝑦). In many problems, such as non-negative matrix factorization, one is interested in less extreme factorizations; specifically, one wishes to compare 𝑝(𝑥,𝑦) to a low-rank matrix approximation in some unknown variable 𝑤; that is, to what degree one might have
-互信息的KL散度公式是基于这样一个结论的：人们会更关注将<math>p(x,y)</math>与完全分解的'''<font color="#ff8000">外积 Outer Product</font>'''<math>p(x) \cdot p(y)</math>进行比较。在许多问题中，例如'''<font color="#ff8000">非负矩阵因式分解 Non-negative matrix factorization</font>'''，人们对较不极端的因式分解感兴趣；具体地说，人们希望将<math>p(x,y)</math>与某个未知变量<math>w</math>中的低秩矩阵近似进行比较；也就是说，在多大程度上可能会有这样的结果：
+互信息的KL散度公式是基于这样一个结论的：人们会更关注将<math>p(x,y)</math>与完全分解的'''<font color="#ff8000">外积 Outer Product</font>'''<math>p(x) \cdot p(y)</math>进行比较。在许多问题中，例如'''<font color="#ff8000">非负矩阵因式分解 Non-negative matrix factorization</font>'''，人们对非极端因式分解感兴趣；具体地说，人们希望将<math>p(x,y)</math>与某个未知变量<math>w</math>中的低秩矩阵近似进行比较；也就是说，在多大程度上可能会有这样的结果：
 :<math>p(x,y)\approx \sum_w p^\prime (x,w) p^{\prime\prime}(w,y)</math>

←上一版本		2021年2月7日 (日) 09:38的版本
第310行：		第310行：


−	~~最近的一篇论文~~<ref>{{cite journal \| last1 = Wolpert \| first1 = D.H \| last2 = DeDeo \| first2 = S. \| year = 2013 \| title = Estimating Functions of Distributions Defined over Spaces of Unknown Size \| journal = Entropy \| volume = 15 \| issue = 12 \| pages = 4668–4699 \| doi = 10.3390/e15114668 \| arxiv = 1311.4548 \| bibcode = 2013Entrp..15.4668W }}</ref>~~中基于一个专门针对互信息本身估计的先验知识。~~	+	请参阅<ref>{{cite journal \| last1 = Wolpert \| first1 = D.H \| last2 = DeDeo \| first2 = S. \| year = 2013 \| title = Estimating Functions of Distributions Defined over Spaces of Unknown Size \| journal = Entropy \| volume = 15 \| issue = 12 \| pages = 4668–4699 \| doi = 10.3390/e15114668 \| arxiv = 1311.4548 \| bibcode = 2013Entrp..15.4668W }}</ref>最近的一篇论文，该论文基于先验知识对互信息本身进行估计。

←上一版本		2021年2月7日 (日) 09:11的版本
第302行：		第302行：


−	~~关于这方面的第一项工作~~<ref>{{cite journal \| last1 = Wolpert \| first1 = D.H. \| last2 = Wolf \| first2 = D.R. \| year = 1995 \| title = Estimating functions of probability distributions from a finite set of samples \| journal = Physical Review E \| volume = 52 \| issue = 6 \| pages = 6841–6854 \| doi = 10.1103/PhysRevE.52.6841 \| pmid = 9964199 \| citeseerx = 10.1.1.55.7122 \| bibcode = 1995PhRvE..52.6841W }}</ref>~~,它还展示了如何对贝叶斯估计进行除互信息之外的许多其他信息理论性质。后来的研究人员重新推导了~~<ref>{{cite journal \| last1 = Hutter \| first1 = M. \| year = 2001 \| title = Distribution of Mutual Information \| journal = Advances in Neural Information Processing Systems 2001 }}</ref>~~内容，并扩展了~~<ref>{{cite journal \| last1 = Archer \| first1 = E. \| last2 = Park \| first2 = I.M. \| last3 = Pillow \| first3 = J. \| year = 2013 \| title = Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data \| journal = Entropy\| volume = 15 \| issue = 12 \| pages = 1738–1755 \| doi = 10.3390/e15051738 \| citeseerx = 10.1.1.294.4690 \| bibcode = 2013Entrp..15.1738A }}</ref>分析。	+	这方面的第一项工作<ref>{{cite journal \| last1 = Wolpert \| first1 = D.H. \| last2 = Wolf \| first2 = D.R. \| year = 1995 \| title = Estimating functions of probability distributions from a finite set of samples \| journal = Physical Review E \| volume = 52 \| issue = 6 \| pages = 6841–6854 \| doi = 10.1103/PhysRevE.52.6841 \| pmid = 9964199 \| citeseerx = 10.1.1.55.7122 \| bibcode = 1995PhRvE..52.6841W }}</ref>也展示了如何对除互信息之外的许多其他信息理论性质进行贝叶斯估计。后来的研究人员重新推导了<ref>{{cite journal \| last1 = Hutter \| first1 = M. \| year = 2001 \| title = Distribution of Mutual Information \| journal = Advances in Neural Information Processing Systems 2001 }}</ref>这一内容，并进行了扩展<ref>{{cite journal \| last1 = Archer \| first1 = E. \| last2 = Park \| first2 = I.M. \| last3 = Pillow \| first3 = J. \| year = 2013 \| title = Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data \| journal = Entropy\| volume = 15 \| issue = 12 \| pages = 1738–1755 \| doi = 10.3390/e15051738 \| citeseerx = 10.1.1.294.4690 \| bibcode = 2013Entrp..15.1738A }}</ref>分析。

@@ 第296行： / 第296行： @@
 It is well-understood how to do Bayesian estimation of the mutual information of a joint distribution based on samples of that distribution.
-如何根据联合分布的样本对联合分布的互信息进行贝叶斯估计，是一个众所周知的问题
+如何根据联合分布的样本对联合分布的互信息进行贝叶斯估计，是很容易理解的。
 The first work to do this, which also showed how to do Bayesian estimation of many other information-theoretic properties besides mutual information, was <ref>{{cite journal | last1 = Wolpert | first1 = D.H. | last2 = Wolf | first2 = D.R. | year = 1995 | title = Estimating functions of probability distributions from a finite set of samples | journal = Physical Review E | volume = 52 | issue = 6 | pages = 6841–6854 | doi = 10.1103/PhysRevE.52.6841 | pmid = 9964199 | citeseerx = 10.1.1.55.7122 | bibcode = 1995PhRvE..52.6841W }}</ref>. Subsequent researchers have rederived <ref>{{cite journal | last1 = Hutter | first1 = M. | year = 2001 | title = Distribution of Mutual Information | journal = Advances in Neural Information Processing Systems 2001 }}</ref>

@@ 第63行： / 第63行： @@
-Notice, as per property of the [[Kullback–Leibler divergence]], that <math>I(X;Y)</math> is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when <math>X</math> and <math>Y</math> are independent (and hence observing <math>Y</math> tells you nothing about <math>X</math>). '''<font color="#32CD32">In general <math>I(X;Y)</math> is non-negative, it is a measure of the price for encoding <math>(X,Y)</math> as a pair of independent random variables, when in reality they are not.</font>'''
+Notice, as per property of the [[Kullback–Leibler divergence]], that <math>I(X;Y)</math> is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when <math>X</math> and <math>Y</math> are independent (and hence observing <math>Y</math> tells you nothing about <math>X</math>). In general <math>I(X;Y)</math> is non-negative, it is a measure of the price for encoding <math>(X,Y)</math> as a pair of independent random variables, when in reality they are not.
-需要注意的是，根据KL散度的性质，当两个随机变量的联合分布与其分别的边缘分布的乘积相等时，如当<math>X</math>和<math>Y</math>是相互独立时（因此观察y不会得到x的信息），<math>I(X;Y)</math>等于零（因此已知<math>Y</math>的信息并不能得到任何关于<math>X</math>的信息）。'''<font color="#32CD32">一般来说，<math>I(X;Y)</math>是非负的，因为它是将<math>(X,Y)</math>作为一对独立随机变量来编码进而进行价值度量的，但实际上它们并不一定是非负的。</font>'''
+需要注意的是，根据KL散度的性质，当两个随机变量的联合分布与其分别的边缘分布的乘积相等时，如当<math>X</math>和<math>Y</math>是相互独立时（因此观察y不会得到x的信息），<math>I(X;Y)</math>等于零（因此已知<math>Y</math>的信息并不能得到任何关于<math>X</math>的信息）。一般来说，<math>I(X;Y)</math>是非负的，因为它是将<math>(X,Y)</math>作为一对独立随机变量来编码进而进行价值度量的，但实际上它们并不一定是非负的。
 == 关于离散分布的PMF In terms of PMFs for discrete distributions ==