# 互信息

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables $\displaystyle{ X }$ and $\displaystyle{ Y }$. The area contained by both circles is the [[joint entropy 这里的维恩图显示了各种信息间的交并补运算关系，这些信息都可以用来度量变量$\displaystyle{ X }$$\displaystyle{ Y }$的各种相关性。图中所有面积（包括两个圆圈）表示二者的 联合熵 Joint entropy$\displaystyle{ H(X,Y) }$。左侧的整个圆圈表示变量$\displaystyle{ X }$ 独立熵 Individual entropy$\displaystyle{ H(X) }$，红色（差集）部分表示X的 条件熵 Conditional entropy$\displaystyle{ H(X|Y) }$。右侧的整个圆圈表示变量$\displaystyle{ Y }$的独立熵$\displaystyle{ H(Y) }$，蓝色（差集）部分表示X的条件熵$\displaystyle{ H(Y|X) }$。两个圆中间的交集部分（紫色的部分）表示二者的互信息 Mutual information，（MI）$\displaystyle{ \operatorname{I}(X;Y) }$）。]] --趣木木讨论）图片应该按照[图1：英文+中文] $\displaystyle{ H(X,Y) }$. The circle on the left (red and violet) is the individual entropy $\displaystyle{ H(X) }$, with the red being the conditional entropy $\displaystyle{ H(X|Y) }$. The circle on the right (blue and violet) is $\displaystyle{ H(Y) }$, with the blue being $\displaystyle{ H(Y|X) }$. The violet is the mutual information $\displaystyle{ \operatorname{I}(X;Y) }$. 这里的维恩图显示了各种信息间的交并补运算关系，这些信息都可以用来度量变量$\displaystyle{ X }$$\displaystyle{ Y }$的各种相关性。图中所有面积（包括两个圆圈）表示二者的 联合熵 Joint entropy$\displaystyle{ H(X,Y) }$。左侧的整个圆圈表示变量$\displaystyle{ X }$ 独立熵 Individual entropy$\displaystyle{ H(X) }$，红色（差集）部分表示X的 条件熵 Conditional entropy$\displaystyle{ H(X|Y) }$。右侧的整个圆圈表示变量$\displaystyle{ Y }$的独立熵$\displaystyle{ H(Y) }$，蓝色（差集）部分表示X的条件熵$\displaystyle{ H(Y|X) }$。两个圆中间的交集部分（紫色的部分）表示二者的互信息 Mutual information（MI） $\displaystyle{ \operatorname{I}(X;Y) }$）。

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables $\displaystyle{ X }$ and $\displaystyle{ Y }$. The area contained by both circles is the joint entropy $\displaystyle{ H(X,Y) }$. The circle on the left (red and violet) is the individual entropy $\displaystyle{ H(X) }$, with the red being the conditional entropy $\displaystyle{ H(X|Y) }$. The circle on the right (blue and violet) is $\displaystyle{ H(Y) }$, with the blue being $\displaystyle{ H(Y|X) }$. The violet is the mutual information $\displaystyle{ \operatorname{I}(X;Y) }$.

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables 𝑋 and 𝑌. The area contained by both circles is the joint entropy H(𝑋,𝑌). The circle on the left (red and violet) is the individual entropy H(𝑋), with the red being the conditional entropy H(𝑋|𝑌). The circle on the right (blue and violet) is H(𝑌), with the blue being H(𝑌|𝑋). The violet is the mutual information I(𝑋;𝑌).

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons, commonly called bits) obtained about one random variable through observing the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair $\displaystyle{ (X,Y) }$ is to the product of the marginal distributions of $\displaystyle{ X }$ and $\displaystyle{ Y }$. MI is the expected value of the pointwise mutual information (PMI).

Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair $\displaystyle{ (X,Y) }$ is to the product of the marginal distributions of $\displaystyle{ X }$ and $\displaystyle{ Y }$. MI is the expected value of the pointwise mutual information (PMI).

Mutual Information is also known as information gain.

Mutual Information is also known as information gain.

## 定义 Definition

Let $\displaystyle{ (X,Y) }$ be a pair of random variables with values over the space $\displaystyle{ \mathcal{X}\times\mathcal{Y} }$. If their joint distribution is $\displaystyle{ P_{(X,Y)} }$ and the marginal distributions are $\displaystyle{ P_X }$ and $\displaystyle{ P_Y }$, the mutual information is defined as

Let $\displaystyle{ (X,Y) }$ be a pair of random variables with values over the space $\displaystyle{ \mathcal{X}\times\mathcal{Y} }$. If their joint distribution is $\displaystyle{ P_{(X,Y)} }$ and the marginal distributions are $\displaystyle{ P_X }$ and $\displaystyle{ P_Y }$, the mutual information is defined as

where $\displaystyle{ D_{\mathrm{KL}} }$ is the Kullback–Leibler divergence.

Notice, as per property of the Kullback–Leibler divergence, that $\displaystyle{ I(X;Y) }$ is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when $\displaystyle{ X }$ and $\displaystyle{ Y }$ are independent (and hence observing $\displaystyle{ Y }$ tells you nothing about $\displaystyle{ X }$). In general $\displaystyle{ I(X;Y) }$ is non-negative, it is a measure of the price for encoding $\displaystyle{ (X,Y) }$ as a pair of independent random variables, when in reality they are not.

## 关于离散分布的PMF In terms of PMFs for discrete distributions

The mutual information of two jointly discrete random variables $\displaystyle{ X }$ and $\displaystyle{ Y }$ is calculated as a double sum:[1]:20

The mutual information of two jointly discrete random variables $\displaystyle{ X }$ and $\displaystyle{ Y }$ is calculated as a double sum:

where $\displaystyle{ p_{(X,Y)} }$ is the joint probability mass function of $\displaystyle{ X }$ and $\displaystyle{ Y }$, and $\displaystyle{ p_X }$ and $\displaystyle{ p_Y }$ are the marginal probability mass functions of $\displaystyle{ X }$ and $\displaystyle{ Y }$ respectively.

where $\displaystyle{ p_{(X,Y)} }$ is the joint probability mass function of $\displaystyle{ X }$ and $\displaystyle{ Y }$, and $\displaystyle{ p_X }$ and $\displaystyle{ p_Y }$ are the marginal probability mass functions of $\displaystyle{ X }$ and $\displaystyle{ Y }$ respectively.

$\displaystyle{ p_{(X,Y)} }$$\displaystyle{ X }$$\displaystyle{ Y }$联合概率质量函数 Probability Mass Functions，而$\displaystyle{ p_X }$$\displaystyle{ p_Y }$分别是数学$\displaystyle{ X }$$\displaystyle{ Y }$边缘概率质量函数 Marginal Probability Mass Functions

## 连续分布的PDF In terms of PDFs for continuous distributions

In the case of jointly continuous random variables, the double sum is replaced by a double integral:[1]:251

In the case of jointly continuous random variables, the double sum is replaced by a double integral:

where $\displaystyle{ p_{(X,Y)} }$ is now the joint probability density function of $\displaystyle{ X }$ and $\displaystyle{ Y }$, and $\displaystyle{ p_X }$ and $\displaystyle{ p_Y }$ are the marginal probability density functions of $\displaystyle{ X }$ and $\displaystyle{ Y }$ respectively.

where $\displaystyle{ p_{(X,Y)} }$ is now the joint probability density function of $\displaystyle{ X }$ and $\displaystyle{ Y }$, and $\displaystyle{ p_X }$ and $\displaystyle{ p_Y }$ are the marginal probability density functions of $\displaystyle{ X }$ and $\displaystyle{ Y }$ respectively.

If the log base 2 is used, the units of mutual information are bits.

If the log base 2 is used, the units of mutual information are bits.

## 动机 Motivation

Intuitively, mutual information measures the information that $\displaystyle{ X }$ and $\displaystyle{ Y }$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if $\displaystyle{ X }$ and $\displaystyle{ Y }$ are independent, then knowing $\displaystyle{ X }$ does not give any information about $\displaystyle{ Y }$ and vice versa, so their mutual information is zero. At the other extreme, if $\displaystyle{ X }$ is a deterministic function of $\displaystyle{ Y }$ and $\displaystyle{ Y }$ is a deterministic function of $\displaystyle{ X }$ then all information conveyed by $\displaystyle{ X }$ is shared with $\displaystyle{ Y }$: knowing $\displaystyle{ X }$ determines the value of $\displaystyle{ Y }$ and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in $\displaystyle{ Y }$ (or $\displaystyle{ X }$) alone, namely the entropy of $\displaystyle{ Y }$ (or $\displaystyle{ X }$). Moreover, this mutual information is the same as the entropy of $\displaystyle{ X }$ and as the entropy of $\displaystyle{ Y }$. (A very special case of this is when $\displaystyle{ X }$ and $\displaystyle{ Y }$ are the same random variable.)

Intuitively, mutual information measures the information that $\displaystyle{ X }$ and $\displaystyle{ Y }$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if $\displaystyle{ X }$ and $\displaystyle{ Y }$ are independent, then knowing $\displaystyle{ X }$ does not give any information about $\displaystyle{ Y }$ and vice versa, so their mutual information is zero. At the other extreme, if $\displaystyle{ X }$ is a deterministic function of $\displaystyle{ Y }$ and $\displaystyle{ Y }$ is a deterministic function of $\displaystyle{ X }$ then all information conveyed by $\displaystyle{ X }$ is shared with $\displaystyle{ Y }$: knowing $\displaystyle{ X }$ determines the value of $\displaystyle{ Y }$ and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in $\displaystyle{ Y }$ (or $\displaystyle{ X }$) alone, namely the entropy of $\displaystyle{ Y }$ (or $\displaystyle{ X }$). Moreover, this mutual information is the same as the entropy of $\displaystyle{ X }$ and as the entropy of $\displaystyle{ Y }$. (A very special case of this is when $\displaystyle{ X }$ and $\displaystyle{ Y }$ are the same random variable.)

Mutual information is a measure of the inherent dependence expressed in the joint distribution of $\displaystyle{ X }$ and $\displaystyle{ Y }$ relative to the joint distribution of $\displaystyle{ X }$ and $\displaystyle{ Y }$ under the assumption of independence. Mutual information therefore measures dependence in the following sense: $\displaystyle{ \operatorname{I}(X;Y)=0 }$ if and only if $\displaystyle{ X }$ and $\displaystyle{ Y }$ are independent random variables. This is easy to see in one direction: if $\displaystyle{ X }$ and $\displaystyle{ Y }$ are independent, then $\displaystyle{ p_{(X,Y)}(x,y)=p_X(x) \cdot p_Y(y) }$, and therefore:

Mutual information is a measure of the inherent dependence expressed in the joint distribution of 𝑋 and 𝑌 relative to the joint distribution of 𝑋 and 𝑌 under the assumption of independence. Mutual information therefore measures dependence in the following sense: I(𝑋;𝑌)=0 if and only if 𝑋 and 𝑌 are independent random variables. This is easy to see in one direction: if 𝑋 and 𝑌 are independent, then 𝑝(𝑋,𝑌)(𝑥,𝑦)=𝑝𝑋(𝑥)⋅𝑝𝑌(𝑦), and therefore:

-- flipped（[[用户讨论: flipped |第一句话有一点点不理解 in the joint distribution of $\displaystyle{ X }$ and $\displaystyle{ Y }$ relative to the joint distribution of $\displaystyle{ X }$ and $\displaystyle{ Y }$]]）

$\displaystyle{ \log{ \left( \frac{p_{(X,Y)}(x,y)}{p_X(x)\,p_Y(y)} \right) } = \log 1 = 0 . }$

Moreover, mutual information is nonnegative (i.e. $\displaystyle{ \operatorname{I}(X;Y) \ge 0 }$ see below) and symmetric (i.e. $\displaystyle{ \operatorname{I}(X;Y) = \operatorname{I}(Y;X) }$ see below).

Moreover, mutual information is nonnegative (i.e. $\displaystyle{ \operatorname{I}(X;Y) \ge 0 }$ see below) and symmetric (i.e. $\displaystyle{ \operatorname{I}(X;Y) = \operatorname{I}(Y;X) }$ see below).

## 与其他量的关系 Relation to other quantities

### 非负性 Nonnegativity

Using Jensen's inequality on the definition of mutual information we can show that $\displaystyle{ \operatorname{I}(X;Y) }$ is non-negative, i.e.[1]:28

Using Jensen's inequality on the definition of mutual information we can show that $\displaystyle{ \operatorname{I}(X;Y) }$ is non-negative, i.e.

$\displaystyle{ \operatorname{I}(X;Y) \ge 0 }$

### 对称性 Symmetry

$\displaystyle{ \operatorname{I}(X;Y) = \operatorname{I}(Y;X) }$

### 条件熵与联合熵的关系 Relation to conditional and joint entropy

Mutual information can be equivalently expressed as:

Mutual information can be equivalently expressed as:

where $\displaystyle{ H(X) }$ and $\displaystyle{ H(Y) }$ are the marginal entropies, $\displaystyle{ H(X|Y) }$ and $\displaystyle{ H(Y|X) }$ are the conditional entropies, and $\displaystyle{ H(X,Y) }$ is the joint entropy of $\displaystyle{ X }$ and $\displaystyle{ Y }$.

Notice the analogy to the union, difference, and intersection of two sets: in this respect, all the formulas given above are apparent from the Venn diagram reported at the beginning of the article.

In terms of a communication channel in which the output $\displaystyle{ Y }$ is a noisy version of the input $\displaystyle{ X }$, these relations are summarised in the figure:

The relationships between information theoretic quantities 信息论量之间的关系

Because $\displaystyle{ \operatorname{I}(X;Y) }$ is non-negative, consequently, $\displaystyle{ H(X) \ge H(X|Y) }$. Here we give the detailed deduction of $\displaystyle{ \operatorname{I}(X;Y)=H(Y)-H(Y|X) }$ for the case of jointly discrete random variables:

The proofs of the other identities above are similar. The proof of the general case (not just discrete) is similar, with integrals replacing sums.

Intuitively, if entropy $\displaystyle{ H(Y) }$ is regarded as a measure of uncertainty about a random variable, then $\displaystyle{ H(Y|X) }$ is a measure of what $\displaystyle{ X }$ does not say about $\displaystyle{ Y }$. This is "the amount of uncertainty remaining about $\displaystyle{ Y }$ after $\displaystyle{ X }$ is known", and thus the right side of the second of these equalities can be read as "the amount of uncertainty in $\displaystyle{ Y }$, minus the amount of uncertainty in $\displaystyle{ Y }$ which remains after $\displaystyle{ X }$ is known", which is equivalent to "the amount of uncertainty in $\displaystyle{ Y }$ which is removed by knowing $\displaystyle{ X }$". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.

Intuitively, if entropy 𝐻(𝑌) is regarded as a measure of uncertainty about a random variable, then 𝐻(𝑌|𝑋) is a measure of what 𝑋 does not say about 𝑌. This is "the amount of uncertainty remaining about 𝑌 after 𝑋 is known", and thus the right side of the second of these equalities can be read as "the amount of uncertainty in 𝑌, minus the amount of uncertainty in 𝑌 which remains after 𝑋 is known", which is equivalent to "the amount of uncertainty in 𝑌 which is removed by knowing 𝑋". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.

Note that in the discrete case $\displaystyle{ H(X|X) = 0 }$ and therefore $\displaystyle{ H(X) = \operatorname{I}(X;X) }$. Thus $\displaystyle{ \operatorname{I}(X; X) \ge \operatorname{I}(X; Y) }$, and one can formulate the basic principle that a variable contains at least as much information about itself as any other variable can provide.

### 与相对熵的关系 Relation to Kullback–Leibler divergence

For jointly discrete or jointly continuous pairs $\displaystyle{ (X,Y) }$,

For jointly discrete or jointly continuous pairs $\displaystyle{ (X,Y) }$,

mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, $\displaystyle{ p_X \cdot p_Y }$, from the joint distribution $\displaystyle{ p_{(X,Y)} }$, that is,

mutual information is the Kullback–Leibler divergence of the product of the marginal distributions, 𝑝𝑋⋅𝑝𝑌, from the joint distribution 𝑝(𝑋,𝑌), that is,

Furthermore, let $\displaystyle{ p_{X|Y=y}(x) = p_{(X,Y)}(x,y) / p_Y(y) }$ be the conditional mass or density function. Then, we have the identity

Furthermore, let $\displaystyle{ p_{X|Y=y}(x) = p_{(X,Y)}(x,y) / p_Y(y) }$ be the conditional mass or density function. Then, we have the identity

The proof for jointly discrete random variables is as follows:

The proof for jointly discrete random variables is as follows:

Similarly this identity can be established for jointly continuous random variables.

Similarly this identity can be established for jointly continuous random variables.

Note that here the Kullback–Leibler divergence involves integration over the values of the random variable $\displaystyle{ X }$ only, and the expression $\displaystyle{ D_\text{KL}(p_{X|Y} \parallel p_X) }$ still denotes a random variable because $\displaystyle{ Y }$ is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution $\displaystyle{ p_X }$ of $\displaystyle{ X }$ from the conditional distribution $\displaystyle{ p_{X|Y} }$ of $\displaystyle{ X }$ given $\displaystyle{ Y }$: the more different the distributions $\displaystyle{ p_{X|Y} }$ and $\displaystyle{ p_X }$ are on average, the greater the information gain.

Note that here the Kullback–Leibler divergence involves integration over the values of the random variable $\displaystyle{ X }$ only, and the expression $\displaystyle{ D_\text{KL}(p_{X|Y} \parallel p_X) }$ still denotes a random variable because $\displaystyle{ Y }$ is random. Thus mutual information can also be understood as the expectation of the Kullback–Leibler divergence of the univariate distribution $\displaystyle{ p_X }$ of $\displaystyle{ X }$ from the conditional distribution $\displaystyle{ p_{X|Y} }$ of $\displaystyle{ X }$ given $\displaystyle{ Y }$: the more different the distributions $\displaystyle{ p_{X|Y} }$ and $\displaystyle{ p_X }$ are on average, the greater the information gain.

### 互信息的贝叶斯估计 Bayesian estimation of mutual information

It is well-understood how to do Bayesian estimation of the mutual information of a joint distribution based on samples of that distribution.

It is well-understood how to do Bayesian estimation of the mutual information of a joint distribution based on samples of that distribution.

The first work to do this, which also showed how to do Bayesian estimation of many other information-theoretic properties besides mutual information, was [2]. Subsequent researchers have rederived [3] and extended [4]this analysis.

See [8]for a recent paper based on a prior specifically tailored to estimation of mutual information per se.

Besides, recently an estimation method accounting for continuous and multivariate outputs, $\displaystyle{ Y }$, was proposed in [10].

### 独立性假设 Independence assumptions

The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing $\displaystyle{ p(x,y) }$ to the fully factorized outer product $\displaystyle{ p(x) \cdot p(y) }$. In many problems, such as non-negative matrix factorization, one is interested in less extreme factorizations; specifically, one wishes to compare $\displaystyle{ p(x,y) }$ to a low-rank matrix approximation in some unknown variable $\displaystyle{ w }$; that is, to what degree one might have

The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing 𝑝(𝑥,𝑦) to the fully factorized outer product 𝑝(𝑥)⋅𝑝(𝑦). In many problems, such as non-negative matrix factorization, one is interested in less extreme factorizations; specifically, one wishes to compare 𝑝(𝑥,𝑦) to a low-rank matrix approximation in some unknown variable 𝑤; that is, to what degree one might have

$\displaystyle{ p(x,y)\approx \sum_w p^\prime (x,w) p^{\prime\prime}(w,y) }$

Alternately, one might be interested in knowing how much more information $\displaystyle{ p(x,y) }$ carries over its factorization. In such a case, the excess information that the full distribution $\displaystyle{ p(x,y) }$ carries over the matrix factorization is given by the Kullback-Leibler divergence

Alternately, one might be interested in knowing how much more information 𝑝(𝑥,𝑦) carries over its factorization. In such a case, the excess information that the full distribution 𝑝(𝑥,𝑦) carries over the matrix factorization is given by the Kullback-Leibler divergence

$\displaystyle{ \operatorname{I}_{LRMA} = \sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}} {p(x,y) \log{ \left(\frac{p(x,y)}{\sum_w p^\prime (x,w) p^{\prime\prime}(w,y)} \right) }}, }$

The conventional definition of the mutual information is recovered in the extreme case that the process $\displaystyle{ W }$ has only one value for $\displaystyle{ w }$.

The conventional definition of the mutual information is recovered in the extreme case that the process $\displaystyle{ W }$ has only one value for $\displaystyle{ w }$.

## 变种 Variations

Several variations on mutual information have been proposed to suit various needs. Among these are normalized variants and generalizations to more than two variables.

Several variations on mutual information have been proposed to suit various needs. Among these are normalized variants and generalizations to more than two variables.

### 度量 Metric

Many applications require a metric, that is, a distance measure between pairs of points. The quantity

Many applications require a metric, that is, a distance measure between pairs of points. The quantity

\displaystyle{ \begin{align} d(X,Y) &= H(X,Y) - \operatorname{I}(X;Y) \\ &= H(X) + H(Y) - 2\operatorname{I}(X;Y) \\ &= H(X|Y) + H(Y|X) \end{align} }

satisfies the properties of a metric (triangle inequality, non-negativity, indiscernability and symmetry). This distance metric is also known as the variation of information.

satisfies the properties of a metric (triangle inequality, non-negativity, indiscernability and symmetry). This distance metric is also known as the variation of information.

If $\displaystyle{ X, Y }$ are discrete random variables then all the entropy terms are non-negative, so $\displaystyle{ 0 \le d(X,Y) \le H(X,Y) }$ and one can define a normalized distance

If 𝑋,𝑌 are discrete random variables then all the entropy terms are non-negative, so 0≤𝑑(𝑋,𝑌)≤𝐻(𝑋,𝑌) and one can define a normalized distance

$\displaystyle{ D(X,Y) = \frac{d(X, Y)}{H(X, Y)} \le 1. }$

The metric $\displaystyle{ D }$ is a universal metric, in that if any other distance measure places $\displaystyle{ X }$ and $\displaystyle{ Y }$ close-by, then the $\displaystyle{ D }$ will also judge them close.[12]模板:Dubious

The metric 𝐷 is a universal metric, in that if any other distance measure places 𝑋 and 𝑌 close-by, then the 𝐷 will also judge them close.

Plugging in the definitions shows that

Plugging in the definitions shows that

$\displaystyle{ D(X,Y) = 1 - \frac{\operatorname{I}(X; Y)}{H(X, Y)}. }$

In a set-theoretic interpretation of information (see the figure for Conditional entropy), this is effectively the Jaccard distance between $\displaystyle{ X }$ and $\displaystyle{ Y }$.

In a set-theoretic interpretation of information (see the figure for Conditional entropy), this is effectively the Jaccard distance between 𝑋 and 𝑌.

Finally,

Finally,

$\displaystyle{ D^\prime(X, Y) = 1 - \frac{\operatorname{I}(X; Y)}{\max\left\{H(X), H(Y)\right\}} }$

is also a metric.

is also a metric.

### 条件互信息 Conditional mutual information

Sometimes it is useful to express the mutual information of two random variables conditioned on a third.

Sometimes it is useful to express the mutual information of two random variables conditioned on a third.

$\displaystyle{ \operatorname{I}(X;Y|Z) = \mathbb{E}_Z [D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} )] }$

For jointly discrete random variables this takes the form

For jointly discrete random variables this takes the form

$\displaystyle{ \operatorname{I}(X;Y|Z) = \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} {p_Z(z)\, p_{X,Y|Z}(x,y|z) \log\left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}\,(x|z)p_{Y|Z}(y|z)}\right]}, }$

which can be simplified as

which can be simplified as

$\displaystyle{ \operatorname{I}(X;Y|Z) = \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}} p_{X,Y,Z}(x,y,z) \log \frac{p_{X,Y,Z}(x,y,z)p_{Z}(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}. }$

For jointly continuous random variables this takes the form

For jointly continuous random variables this takes the form

$\displaystyle{ \operatorname{I}(X;Y|Z) = \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}} {p_Z(z)\, p_{X,Y|Z}(x,y|z) \log\left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}\,(x|z)p_{Y|Z}(y|z)}\right]} dx dy dz, }$

which can be simplified as

which can be simplified as

$\displaystyle{ \operatorname{I}(X;Y|Z) = \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}} p_{X,Y,Z}(x,y,z) \log \frac{p_{X,Y,Z}(x,y,z)p_{Z}(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)} dx dy dz. }$

Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that

Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that

$\displaystyle{ \operatorname{I}(X;Y|Z) \ge 0 }$

for discrete, jointly distributed random variables $\displaystyle{ X,Y,Z }$. This result has been used as a basic building block for proving other inequalities in information theory.

for discrete, jointly distributed random variables $\displaystyle{ X,Y,Z }$. This result has been used as a basic building block for proving other inequalities in information theory.

### 多元互信息 Multivariate mutual information

Several generalizations of mutual information to more than two random variables have been proposed, such as total correlation (or multi-information) and interaction information. The expression and study of multivariate higher-degree mutual-information was achieved in two seemingly independent works: McGill (1954) [14] who called these functions “interaction information”, and Hu Kuo Ting (1962) [15] who also first proved the possible negativity of mutual-information for degrees higher than 2 and justified algebraically the intuitive correspondence to Venn diagrams [16]

Several generalizations of mutual information to more than two random variables have been proposed, such as total correlation (or multi-information) and interaction information. The expression and study of multivariate higher-degree mutual-information was achieved in two seemingly independent works: McGill (1954) who called these functions “interaction information”, and Hu Kuo Ting (1962) who also first proved the possible negativity of mutual-information for degrees higher than 2 and justified algebraically the intuitive correspondence to Venn diagrams

$\displaystyle{ \operatorname{I}(X_1;X_1) = H(X_1) }$

and for $\displaystyle{ n \gt 1, }$

and for 𝑛>1,

$\displaystyle{ \operatorname{I}(X_1;\,...\,;X_n) = \operatorname{I}(X_1;\,...\,;X_{n-1}) - \operatorname{I}(X_1;\,...\,;X_{n-1}|X_n), }$

where (as above) we define

where (as above) we define

$\displaystyle{ I(X_1;\ldots;X_{n-1}|X_{n}) = \mathbb{E}_{X_{n}} [D_{\mathrm{KL}}( P_{(X_1,\ldots,X_{n-1})|X_{n}} \| P_{X_1|X_{n}} \otimes\cdots\otimes P_{X_{n-1}|X_{n}} )]. }$

(This definition of multivariate mutual information is identical to that of interaction information except for a change in sign when the number of random variables is odd.)

(This definition of multivariate mutual information is identical to that of interaction information except for a change in sign when the number of random variables is odd.)

（这个多元互信息的定义与交互信息的定义相同，对于随机变量的数目为奇数时符号的变化除外。）

#### 多元统计独立性 Multivariate statistical independence

The multivariate mutual-information functions generalize the pairwise independence case that states that $\displaystyle{ X_1,X_2 }$ if and only if $\displaystyle{ I(X_1;X_2)=0 }$, to arbitrary numerous variable. n variables are mutually independent if and only if the $\displaystyle{ 2^n-n-1 }$ mutual information functions vanish $\displaystyle{ I(X_1;...;X_k)=0 }$ with $\displaystyle{ n \ge k \ge 2 }$ (theorem 2 [16]). In this sense, the $\displaystyle{ I(X_1;...;X_k)=0 }$ can be used as a refined statistical independence criterion.

The multivariate mutual-information functions generalize the pairwise independence case that states that 𝑋1,𝑋2 if and only if 𝐼(𝑋1;𝑋2)=0, to arbitrary numerous variable. n variables are mutually independent if and only if the 2𝑛−𝑛−1 mutual information functions vanish 𝐼(𝑋1;...;𝑋𝑘)=0 with 𝑛≥𝑘≥2 (theorem 2). In this sense, the 𝐼(𝑋1;...;𝑋𝑘)=0 can be used as a refined statistical independence criterion.

#### 应用 Applications

For 3 variables, Brenner et al. applied multivariate mutual information to neural coding and called its negativity "synergy" [19] and Watkinson et al. applied it to genetic expression [20]. For arbitrary k variables, Tapia et al. applied multivariate mutual information to gene expression [21] [16]). It can be zero, positive, or negative [22]. The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints [21]).

For 3 variables, Brenner et al. applied multivariate mutual information to neural coding and called its negativity "synergy" and Watkinson et al. applied it to genetic expression . For arbitrary k variables, Tapia et al. applied multivariate mutual information to gene expression . The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints .

One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in feature selection.[25]

One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in feature selection.

Mutual information is also used in the area of signal processing as a measure of similarity between two signals. For example, FMI metric[27] is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The Matlab code for this metric can be found at.[28]

Mutual information is also used in the area of signal processing as a measure of similarity between two signals. For example, FMI metric is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The Matlab code for this metric can be found at.

### 定向信息 Directed information

Directed information, $\displaystyle{ \operatorname{I}\left(X^n \to Y^n\right) }$, measures the amount of information that flows from the process $\displaystyle{ X^n }$ to $\displaystyle{ Y^n }$, where $\displaystyle{ X^n }$ denotes the vector $\displaystyle{ X_1, X_2, ..., X_n }$ and $\displaystyle{ Y^n }$ denotes $\displaystyle{ Y_1, Y_2, ..., Y_n }$. The term directed information was coined by James Massey and is defined as

Directed information, I(𝑋𝑛→𝑌𝑛), measures the amount of information that flows from the process 𝑋𝑛 to 𝑌𝑛, where 𝑋𝑛 denotes the vector 𝑋1,𝑋2,...,𝑋𝑛 and 𝑌𝑛 denotes 𝑌1,𝑌2,...,𝑌𝑛. The term directed information was coined by James Massey and is defined as：

$\displaystyle{ \operatorname{I}\left(X^n \to Y^n\right) = \sum_{i=1}^n \operatorname{I}\left(X^i; Y_i|Y^{i-1}\right) }$.

Note that if $\displaystyle{ n=1 }$, the directed information becomes the mutual information. Directed information has many applications in problems where causality plays an important role, such as capacity of channel with feedback.[31][32]

Note that if 𝑛=1, the directed information becomes the mutual information. Directed information has many applications in problems where causality plays an important role, such as capacity of channel with feedback.

### 归一化变量 Normalized variants

Normalized variants of the mutual information are provided by the coefficients of constraint,模板:Sfn uncertainty coefficient[35] or proficiency:[36]

Normalized variants of the mutual information are provided by the coefficients of constraint, uncertainty coefficient or proficiency:

$\displaystyle{ C_{XY} = \frac{\operatorname{I}(X;Y)}{H(Y)} ~~~~\mbox{和}~~~~ C_{YX} = \frac{\operatorname{I}(X;Y)}{H(X)}. }$

The two coefficients have a value ranging in [0, 1], but are not necessarily equal. In some cases a symmetric measure may be desired, such as the following redundancy[citation needed] measure:

The two coefficients have a value ranging in [0, 1], but are not necessarily equal. In some cases a symmetric measure may be desired, such as the following redundancy measure:

$\displaystyle{ R = \frac{\operatorname{I}(X;Y)}{H(X) + H(Y)} }$

which attains a minimum of zero when the variables are independent and a maximum value of

which attains a minimum of zero when the variables are independent and a maximum value of

$\displaystyle{ R_\max = \frac{\min\left\{H(X), H(Y)\right\}}{H(X) + H(Y)} }$

when one variable becomes completely redundant with the knowledge of the other. See also Redundancy (information theory).

when one variable becomes completely redundant with the knowledge of the other. See also Redundancy (information theory).

Another symmetrical measure is the symmetric uncertainty 模板:Harv, given by

Another symmetrical measure is the symmetric uncertainty , given by

$\displaystyle{ U(X, Y) = 2R = 2\frac{\operatorname{I}(X;Y)}{Ha(X) + H(Y)} }$

which represents the harmonic mean of the two uncertainty coefficients $\displaystyle{ C_{XY}, C_{YX} }$.[35]

which represents the harmonic mean of the two uncertainty coefficients $\displaystyle{ C_{XY}, C_{YX} }$.

If we consider mutual information as a special case of the total correlation or dual total correlation, the normalized version are respectively,

If we consider mutual information as a special case of the total correlation or dual total correlation, the normalized version are respectively,

$\displaystyle{ \frac{\operatorname{I}(X;Y)}{\min\left[ H(X),H(Y)\right]} }$ and $\displaystyle{ \frac{\operatorname{I}(X;Y)}{H(X,Y)} \; . }$

This normalized version also known as Information Quality Ratio (IQR) which quantifies the amount of information of a variable based on another variable against total uncertainty:[37]

This normalized version also known as Information Quality Ratio (IQR) which quantifies the amount of information of a variable based on another variable against total uncertainty:

$\displaystyle{ IQR(X, Y) = \operatorname{E}[\operatorname{I}(X;Y)] = \frac{\operatorname{I}(X;Y)}{H(X, Y)} = \frac{\sum_{x \in X} \sum_{y \in Y} p(x, y) \log {p(x)p(y)}}{\sum_{x \in X} \sum_{y \in Y} p(x, y) \log {p(x, y)}} - 1 }$

There's a normalization[38] which derives from first thinking of mutual information as an analogue to covariance (thus Shannon entropy is analogous to variance). Then the normalized mutual information is calculated akin to the Pearson correlation coefficient,

There's a normalization which derives from first thinking of mutual information as an analogue to covariance (thus Shannon entropy is analogous to variance). Then the normalized mutual information is calculated akin to the Pearson correlation coefficient,

$\displaystyle{ \frac{\operatorname{I}(X;Y)}{\sqrt{H(X)H(Y)}}\; . }$

### 加权变量 Weighted variants

In the traditional formulation of the mutual information,

In the traditional formulation of the mutual information,

$\displaystyle{ \operatorname{I}(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x, y) \log \frac{p(x, y)}{p(x)\,p(y)}, }$

each event or object specified by $\displaystyle{ (x, y) }$ is weighted by the corresponding probability $\displaystyle{ p(x, y) }$. This assumes that all objects or events are equivalent apart from their probability of occurrence. However, in some applications it may be the case that certain objects or events are more significant than others, or that certain patterns of association are more semantically important than others.

each event or object specified by $\displaystyle{ (x, y) }$ is weighted by the corresponding probability $\displaystyle{ p(x, y) }$. This assumes that all objects or events are equivalent apart from their probability of occurrence. However, in some applications it may be the case that certain objects or events are more significant than others, or that certain patterns of association are more semantically important than others.

$\displaystyle{ (x, y) }$ 指定的每个事件或对象都由相应的概率$\displaystyle{ p(x, y) }$进行加权。这假设所有的物体或事件除了发生的概率外都是相等的。然而，在某些应用场景中，某些特定的对象或事件可能比其他对象或事件更重要，或者某些特定的关联模式在语义上比其他模式更重要。

For example, the deterministic mapping $\displaystyle{ \{(1,1),(2,2),(3,3)\} }$ may be viewed as stronger than the deterministic mapping $\displaystyle{ \{(1,3),(2,1),(3,2)\} }$, although these relationships would yield the same mutual information. This is because the mutual information is not sensitive at all to any inherent ordering in the variable values (脚本错误：没有“Footnotes”这个模块。, 脚本错误：没有“Footnotes”这个模块。, 脚本错误：没有“Footnotes”这个模块。), and is therefore not sensitive at all to the form of the relational mapping between the associated variables. If it is desired that the former relation—showing agreement on all variable values—be judged stronger than the later relation, then it is possible to use the following weighted mutual information 模板:Harv.

For example, the deterministic mapping {(1,1),(2,2),(3,3)} may be viewed as stronger than the deterministic mapping {(1,3),(2,1),(3,2)}, although these relationships would yield the same mutual information. This is because the mutual information is not sensitive at all to any inherent ordering in the variable values, and is therefore not sensitive at all to the form of the relational mapping between the associated variables. If it is desired that the former relation—showing agreement on all variable values—be judged stronger than the later relation, then it is possible to use the following weighted mutual information.

$\displaystyle{ \operatorname{I}(X;Y) = \sum_{y \in Y} \sum_{x \in X} w(x,y) p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)}, }$

which places a weight $\displaystyle{ w(x,y) }$ on the probability of each variable value co-occurrence, $\displaystyle{ p(x,y) }$. This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevant holistic or Prägnanz factors. In the above example, using larger relative weights for $\displaystyle{ w(1,1) }$, $\displaystyle{ w(2,2) }$, and $\displaystyle{ w(3,3) }$ would have the effect of assessing greater informativeness for the relation $\displaystyle{ \{(1,1),(2,2),(3,3)\} }$ than for the relation $\displaystyle{ \{(1,3),(2,1),(3,2)\} }$, which may be desirable in some cases of pattern recognition, and the like. This weighted mutual information is a form of weighted KL-Divergence, which is known to take negative values for some inputs,[39] and there are examples where the weighted mutual information also takes negative values.[40]

which places a weight 𝑤(𝑥,𝑦) on the probability of each variable value co-occurrence, 𝑝(𝑥,𝑦). This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevant holistic or Prägnanz factors. In the above example, using larger relative weights for 𝑤(1,1), 𝑤(2,2), and 𝑤(3,3) would have the effect of assessing greater informativeness for the relation {(1,1),(2,2),(3,3)} than for the relation {(1,3),(2,1),(3,2)}, which may be desirable in some cases of pattern recognition, and the like. This weighted mutual information is a form of weighted KL-Divergence, which is known to take negative values for some inputs,[24] and there are examples where the weighted mutual information also takes negative values.[25]

A probability distribution can be viewed as a partition of a set. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? The adjusted mutual information or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to the adjusted Rand index of two different partitions of a set.

A probability distribution can be viewed as a partition of a set. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? The adjusted mutual information or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to the adjusted Rand index of two different partitions of a set.

### 绝对互信息 Absolute mutual information

Using the ideas of Kolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:

Using the ideas of Kolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:

$\displaystyle{ \operatorname{I}_K(X;Y) = K(X) - K(X|Y). }$

To establish that this quantity is symmetric up to a logarithmic factor ($\displaystyle{ \operatorname{I}_K(X;Y) \approx \operatorname{I}_K(Y;X) }$) one requires the chain rule for Kolmogorov complexity 模板:Harvard citation. Approximations of this quantity via compression can be used to define a distance measure to perform a hierarchical clustering of sequences without having any domain knowledge of the sequences 模板:Harvard citation.

To establish that this quantity is symmetric up to a logarithmic factor (I𝐾(𝑋;𝑌)≈I𝐾(𝑌;𝑋)) one requires the chain rule for Kolmogorov complexity.Approximations of this quantity via compression can be used to define a distance measure to perform a hierarchical clustering of sequences without having any domain knowledge of the sequences.

### 线性相关 Linear correlation

Unlike correlation coefficients, such as the product moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for $\displaystyle{ X }$ and $\displaystyle{ Y }$ is a bivariate normal distribution (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between $\displaystyle{ \operatorname{I} }$ and the correlation coefficient $\displaystyle{ \rho }$ 模板:Harv.

Unlike correlation coefficients, such as the product moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for $\displaystyle{ X }$ and $\displaystyle{ Y }$ is a bivariate normal distribution (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between $\displaystyle{ \operatorname{I} }$ and the correlation coefficient $\displaystyle{ \rho }$ .

$\displaystyle{ \operatorname{I} = -\frac{1}{2} \log\left(1 - \rho^2\right) }$

The equation above can be derived as follows for a bivariate Gaussian:

The equation above can be derived as follows for a bivariate Gaussian:

\displaystyle{ \begin{align} \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} &\sim \mathcal{N} \left( \begin{pmatrix} \mu_1 \\ \mu_2 \end{pmatrix}, \Sigma \right),\qquad \Sigma = \begin{pmatrix} \sigma^2_1 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma^2_2 \end{pmatrix} \\ H(X_i) &= \frac{1}{2}\log\left(2\pi e \sigma_i^2\right) = \frac{1}{2} + \frac{1}{2}\log(2\pi) + \log\left(\sigma_i\right), \quad i\in\{1, 2\} \\ H(X_1, X_2) &= \frac{1}{2}\log\left[(2\pi e)^2|\Sigma|\right] = 1 + \log(2\pi) + \log\left(\sigma_1 \sigma_2\right) + \frac{1}{2}\log\left(1 - \rho^2\right) \\ \end{align} }

Therefore,

Therefore,

$\displaystyle{ \operatorname{I}\left(X_1; X_2\right) = H\left(X_1\right) + H\left(X_2\right) - H\left(X_1, X_2\right) = -\frac{1}{2}\log\left(1 - \rho^2\right) }$

### 对于离散数据 For discrete data

When $\displaystyle{ X }$ and $\displaystyle{ Y }$ are limited to be in a discrete number of states, observation data is summarized in a contingency table, with row variable $\displaystyle{ X }$ (or $\displaystyle{ i }$) and column variable $\displaystyle{ Y }$ (or $\displaystyle{ j }$). Mutual information is one of the measures of association or correlation between the row and column variables. Other measures of association include Pearson's chi-squared test statistics, G-test statistics, etc. In fact, mutual information is equal to G-test statistics divided by $\displaystyle{ 2N }$, where $\displaystyle{ N }$ is the sample size.

When 𝑋 and 𝑌 are limited to be in a discrete number of states, observation data is summarized in a contingency table, with row variable 𝑋 (or 𝑖) and column variable 𝑌 (or 𝑗). Mutual information is one of the measures of association or correlation between the row and column variables. Other measures of association include Pearson's chi-squared test statistics, G-test statistics, etc. In fact, mutual information is equal to G-test statistics divided by 2𝑁, where 𝑁 is the sample size.

$\displaystyle{ X }$$\displaystyle{ Y }$被限制为离散状态时，观测数据汇总在列联表 Contingency Table中，其中行变量$\displaystyle{ X }$（或$\displaystyle{ i }$）和列变量$\displaystyle{ Y }$（或$\displaystyle{ j }$）。互信息是行和列变量之间关联或相关性的度量之一。其他关联度量包括Pearson卡方检验统计量、G检验 G-Test统计量等。事实上，互信息等于G检验统计量除以$\displaystyle{ 2N }$，其中$\displaystyle{ N }$为样本量。

## 应用 Applications

In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizing conditional entropy. Examples include:

In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizing conditional entropy. Examples include:

• In search engine technology, mutual information between phrases and contexts is used as a feature for k-means clustering to discover semantic clusters (concepts).[42] For example, the mutual information of a bigram might be calculated as:

$\displaystyle{ MI(x,y) = \log \frac{P_{X,Y}(x,y)}{P_X(x) P_Y(y)} \approx log \frac{\frac{f_{XY}}{B}}{\frac{f_X}{U} \frac{f_Y}{U}} }$

where $\displaystyle{ f_{XY} }$ is the number of times the bigram xy appears in the corpus, $\displaystyle{ f_{X} }$ is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.[42]

where $\displaystyle{ f_{XY} }$ is the number of times the bigram xy appears in the corpus, $\displaystyle{ f_{X} }$ is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.


In telecommunications, the channel capacity is equal to the mutual information, maximized over all input distributions.

Discriminative training procedures for hidden Markov models have been proposed based on the maximum mutual information (MMI) criterion.

RNA secondary structure prediction from a multiple sequence alignment.

Phylogenetic profiling prediction from pairwise present and disappearance of functionally link genes.

Mutual information has been used as a criterion for feature selection and feature transformations in machine learning. It can be used to characterize both the relevance and redundancy of variables, such as the minimum redundancy feature selection.

• Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.

Mutual information is used in determining the similarity of two different clusterings of a dataset. As such, it provides some advantages over the traditional Rand index.

• Mutual information of words is often used as a significance function for the computation of collocations in corpus linguistics. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within $\displaystyle{ N }$ words of another, goes up with $\displaystyle{ N }$.

Mutual information of words is often used as a significance function for the computation of collocations in corpus linguistics. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within 𝑁 words of another, goes up with 𝑁.

• Mutual information is used in medical imaging for image registration. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same coordinate system as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.

Mutual information is used in medical imaging for image registration. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same coordinate system as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.

Detection of phase synchronization in time series analysis

In the infomax method for neural-net and other machine learning, including the infomax-based Independent component analysis algorithm.

Average mutual information in delay embedding theorem is used for determining the embedding delay parameter.

Mutual information between genes in expression microarray data is used by the ARACNE algorithm for reconstruction of gene networks.

ARACNE算法利用表达微阵列数据中基因间的互信息来重构基因网络 Gene Networks

• In statistical mechanics, Loschmidt's paradox may be expressed in terms of mutual information.[43][44] Loschmidt noted that it must be impossible to determine a physical law which lacks time reversal symmetry (e.g. the second law of thermodynamics) only from physical laws which have this symmetry. He pointed out that the H-theorem of Boltzmann made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in phase space, then Liouville's theorem implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by Boltzmann's constant).

In statistical mechanics, Loschmidt's paradox may be expressed in terms of mutual information.[27][28] Loschmidt noted that it must be impossible to determine a physical law which lacks time reversal symmetry (e.g. the second law of thermodynamics) only from physical laws which have this symmetry. He pointed out that the H-theorem of Boltzmann made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in phase space, then Liouville's theorem implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by Boltzmann's constant).

• The mutual information is used to learn the structure of Bayesian networks/dynamic Bayesian networks, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:[45] learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.

The mutual information is used to learn the structure of Bayesian networks/dynamic Bayesian networks, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:[29] learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.

Popular cost function in decision tree learning.

• The mutual information is used in cosmology to test the influence of large-scale environments on galaxy properties in the Galaxy Zoo.

The mutual information is used in cosmology to test the influence of large-scale environments on galaxy properties in the Galaxy Zoo.

• The mutual information was used in Solar Physics to derive the solar differential rotation profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements[47]

The mutual information was used in Solar Physics to derive the solar differential rotation profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements.

• Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.[49]

Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.

## 注释 Notes

1. Cover, T.M.; Thomas, J.A. (1991). Elements of Information Theory (Wiley ed.). ISBN 978-0-471-24195-9.
2. Wolpert, D.H.; Wolf, D.R. (1995). "Estimating functions of probability distributions from a finite set of samples". Physical Review E. 52 (6): 6841–6854. Bibcode:1995PhRvE..52.6841W. CiteSeerX 10.1.1.55.7122. doi:10.1103/PhysRevE.52.6841. PMID 9964199.
3. Hutter, M. (2001). "Distribution of Mutual Information". Advances in Neural Information Processing Systems 2001.
4. Archer, E.; Park, I.M.; Pillow, J. (2013). "Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data". Entropy. 15 (12): 1738–1755. Bibcode:2013Entrp..15.1738A. CiteSeerX 10.1.1.294.4690. doi:10.3390/e15051738.
5. Wolpert, D.H.; Wolf, D.R. (1995). "Estimating functions of probability distributions from a finite set of samples". Physical Review E. 52 (6): 6841–6854. Bibcode:1995PhRvE..52.6841W. CiteSeerX 10.1.1.55.7122. doi:10.1103/PhysRevE.52.6841. PMID 9964199.
6. Hutter, M. (2001). "Distribution of Mutual Information". Advances in Neural Information Processing Systems 2001.
7. Archer, E.; Park, I.M.; Pillow, J. (2013). "Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data". Entropy. 15 (12): 1738–1755. Bibcode:2013Entrp..15.1738A. CiteSeerX 10.1.1.294.4690. doi:10.3390/e15051738.
8. Wolpert, D.H; DeDeo, S. (2013). "Estimating Functions of Distributions Defined over Spaces of Unknown Size". Entropy. 15 (12): 4668–4699. arXiv:1311.4548. Bibcode:2013Entrp..15.4668W. doi:10.3390/e15114668.
9. Wolpert, D.H; DeDeo, S. (2013). "Estimating Functions of Distributions Defined over Spaces of Unknown Size". Entropy. 15 (12): 4668–4699. arXiv:1311.4548. Bibcode:2013Entrp..15.4668W. doi:10.3390/e15114668.
10. Tomasz Jetka; Karol Nienaltowski; Tomasz Winarski; Slawomir Blonski; Michal Komorowski (2019), "Information-theoretic analysis of multivariate single-cell signaling responses", PLOS Computational Biology, 15 (7): e1007132, arXiv:1808.05581, Bibcode:2019PLSCB..15E7132J, doi:10.1371/journal.pcbi.1007132, PMC 6655862, PMID 31299056
11. Tomasz Jetka; Karol Nienaltowski; Tomasz Winarski; Slawomir Blonski; Michal Komorowski (2019), "Information-theoretic analysis of multivariate single-cell signaling responses", PLOS Computational Biology, 15 (7): e1007132, arXiv:1808.05581, Bibcode:2019PLSCB..15E7132J, doi:10.1371/journal.pcbi.1007132, PMC 6655862, PMID 31299056
12. Kraskov, Alexander; Stögbauer, Harald; Andrzejak, Ralph G.; Grassberger, Peter (2003). "Hierarchical Clustering Based on Mutual Information". arXiv:q-bio/0311039. Bibcode:2003q.bio....11039K. Cite journal requires |journal= (help)
13. Kraskov, Alexander; Stögbauer, Harald; Andrzejak, Ralph G.; Grassberger, Peter (2003). "Hierarchical Clustering Based on Mutual Information". arXiv:q-bio/0311039. Bibcode:2003q.bio....11039K. Cite journal requires |journal= (help)
14. McGill, W. (1954). "Multivariate information transmission". Psychometrika. 19 (1): 97–116. doi:10.1007/BF02289159.
15. Hu, K.T. (1962). "On the Amount of Information". Theory Probab. Appl. 7 (4): 439–447. doi:10.1137/1107041.
16. Baudot, P.; Tapia, M.; Bennequin, D.; Goaillard, J.M. (2019). "Topological Information Data Analysis". Entropy. 21 (9). 869. arXiv:1907.04242. Bibcode:2019Entrp..21..869B. doi:10.3390/e21090869.
17. McGill, W. (1954). "Multivariate information transmission". Psychometrika. 19 (1): 97–116. doi:10.1007/BF02289159.
18. Hu, K.T. (1962). "On the Amount of Information". Theory Probab. Appl. 7 (4): 439–447. doi:10.1137/1107041.
19. Brenner, N.; Strong, S.; Koberle, R.; Bialek, W. (2000). "Synergy in a Neural Code". Neural Comput. 12 (7): 1531–1552. doi:10.1162/089976600300015259. PMID 10935917.
20. Watkinson, J.; Liang, K.; Wang, X.; Zheng, T.; Anastassiou, D. (2009). "Inference of Regulatory Gene Interactions from Expression Data Using Three-Way Mutual Information". Chall. Syst. Biol. Ann. N. Y. Acad. Sci. 1158 (1): 302–313. Bibcode:2009NYASA1158..302W. doi:10.1111/j.1749-6632.2008.03757.x. PMID 19348651.
21. Tapia, M.; Baudot, P.; Formizano-Treziny, C.; Dufour, M.; Goaillard, J.M. (2018). "Neurotransmitter identity and electrophysiological phenotype are genetically coupled in midbrain dopaminergic neurons". Sci. Rep. 8 (1): 13637. Bibcode:2018NatSR...813637T. doi:10.1038/s41598-018-31765-z. PMC 6134142. PMID 30206240.
22. Hu, K.T. (1962). "On the Amount of Information". Theory Probab. Appl. 7 (4): 439–447. doi:10.1137/1107041.
23. Brenner, N.; Strong, S.; Koberle, R.; Bialek, W. (2000). "Synergy in a Neural Code". Neural Comput. 12 (7): 1531–1552. doi:10.1162/089976600300015259. PMID 10935917.
24. Watkinson, J.; Liang, K.; Wang, X.; Zheng, T.; Anastassiou, D. (2009). "Inference of Regulatory Gene Interactions from Expression Data Using Three-Way Mutual Information". Chall. Syst. Biol. Ann. N. Y. Acad. Sci. 1158 (1): 302–313. Bibcode:2009NYASA1158..302W. doi:10.1111/j.1749-6632.2008.03757.x. PMID 19348651.
25. Christopher D. Manning; Prabhakar Raghavan; Hinrich Schütze (2008). An Introduction to Information Retrieval. Cambridge University Press. ISBN 978-0-521-86571-5.
26. Christopher D. Manning; Prabhakar Raghavan; Hinrich Schütze (2008). An Introduction to Information Retrieval. Cambridge University Press. ISBN 978-0-521-86571-5.
27. Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012.
28. "Feature Mutual Information (FMI) metric for non-reference image fusion - File Exchange - MATLAB Central". www.mathworks.com. Retrieved 4 April 2018.
29. Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012.
30. "Feature Mutual Information (FMI) metric for non-reference image fusion - File Exchange - MATLAB Central". www.mathworks.com. Retrieved 4 April 2018.
31. Massey, James (1990). "Causality, Feedback And Directed Informatio". Proc. 1990 Intl. Symp. on Info. Th. and its Applications, Waikiki, Hawaii, Nov. 27-30, 1990. CiteSeerX 10.1.1.36.5688.
32. Permuter, Haim Henry; Weissman, Tsachy; Goldsmith, Andrea J. (February 2009). "Finite State Channels With Time-Invariant Deterministic Feedback". IEEE Transactions on Information Theory. 55 (2): 644–662. arXiv:cs/0608070. doi:10.1109/TIT.2008.2009849.
33. Massey, James (1990). "Causality, Feedback And Directed Informatio". Proc. 1990 Intl. Symp. on Info. Th. and its Applications, Waikiki, Hawaii, Nov. 27-30, 1990. CiteSeerX 10.1.1.36.5688.
34. Permuter, Haim Henry; Weissman, Tsachy; Goldsmith, Andrea J. (February 2009). "Finite State Channels With Time-Invariant Deterministic Feedback". IEEE Transactions on Information Theory. 55 (2): 644–662. arXiv:cs/0608070. doi:10.1109/TIT.2008.2009849.
35. Press, WH; Teukolsky, SA; Vetterling, WT; Flannery, BP (2007). "Section 14.7.3. Conditional Entropy and Mutual Information". Numerical Recipes: The Art of Scientific Computing (3rd ed.). New York: Cambridge University Press. ISBN 978-0-521-88068-8.
36. White, Jim; Steingold, Sam; Fournelle, Connie. Performance Metrics for Group-Detection Algorithms (PDF). Interface 2004.
37. Wijaya, Dedy Rahman; Sarno, Riyanarto; Zulaika, Enny (2017). "Information Quality Ratio as a novel metric for mother wavelet selection". Chemometrics and Intelligent Laboratory Systems. 160: 59–71. doi:10.1016/j.chemolab.2016.11.012.
38. Strehl, Alexander; Ghosh, Joydeep (2003). "Cluster Ensembles – A Knowledge Reuse Framework for Combining Multiple Partitions" (PDF). The Journal of Machine Learning Research. 3: 583–617. doi:10.1162/153244303321897735.
39. Kvålseth, T. O. (1991). "The relative useful information measure: some comments". Information Sciences. 56 (1): 35–38. doi:10.1016/0020-0255(91)90022-m.
40. 模板:Cite dissertation
41. 模板:Cite dissertation
42. Parsing a Natural Language Using Mutual Information Statistics by David M. Magerman and Mitchell P. Marcus
43. Hugh Everett Theory of the Universal Wavefunction, Thesis, Princeton University, (1956, 1973), pp 1–140 (page 30)
44. Everett, Hugh (1957). "Relative State Formulation of Quantum Mechanics". Reviews of Modern Physics. 29 (3): 454–462. Bibcode:1957RvMP...29..454E. doi:10.1103/revmodphys.29.454. Archived from the original on 2011-10-27. Retrieved 2012-07-16.
47. Keys, Dustin; Kholikov, Shukur; Pevtsov, Alexei A. (February 2015). "Application of Mutual Information Methods in Time Distance Helioseismology". Solar Physics. 290 (3): 659–671. arXiv:1501.05597. Bibcode:2015SoPh..290..659K. doi:10.1007/s11207-015-0650-y.
48. Keys, Dustin; Kholikov, Shukur; Pevtsov, Alexei A. (February 2015). "Application of Mutual Information Methods in Time Distance Helioseismology". Solar Physics. 290 (3): 659–671. arXiv:1501.05597. Bibcode:2015SoPh..290..659K. doi:10.1007/s11207-015-0650-y.
49. Invariant Information Clustering for Unsupervised Image Classification and Segmentation by Xu Ji, Joao Henriques and Andrea Vedaldi

## 参考资料 References

• Cronbach, L. J. (1954). "On the non-rational application of information measures in psychology". In Quastler, Henry. Information Theory in Psychology: Problems and Methods. Glencoe, Illinois: Free Press. pp. 14–30.

• Coombs, C. H.; Dawes, R. M.; Tversky, A. (1970). Mathematical Psychology: An Elementary Introduction. Englewood Cliffs, New Jersey: Prentice-Hall.

• Gel'fand, I.M.; Yaglom, A.M. (1957). "Calculation of amount of information about a random function contained in another such function". American Mathematical Society Translations: Series 2. 12: 199–246. English translation of original in Uspekhi Matematicheskikh Nauk 12 (1): 3-52.

• Guiasu, Silviu (1977). Information Theory with Applications. McGraw-Hill, New York. ISBN 978-0-07-025109-0.

• Lockhead, G. R. (1970). "Identification and the form of multidimensional discrimination space". Journal of Experimental Psychology. 85 (1): 1–10. doi:10.1037/h0029508. PMID 5458322.

• Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012.

• Athanasios Papoulis. Probability, Random Variables, and Stochastic Processes, second edition. New York: McGraw-Hill, 1984. (See Chapter 15.)

Category:Information theory

Category:Entropy and information

This page was moved from wikipedia:en:Mutual information. Its edit history can be viewed at 互信息/edithistory