联合熵

此词条暂由彩云小译翻译，未经人工整理和审校，带来阅读不便，请见谅。

该图表示在变量X、Y相关联的各种信息量之间，进行加减关系的维恩图。两个圆所包含的区域是联合熵H(X,Y)。左侧的圆圈（红色和紫色）是单个熵H（X），红色是条件熵H（X ǀ Y）。右侧的圆圈（蓝色和紫色）为H（Y），蓝色为H（Y ǀ X）。中间紫色的是相互信息i（X; Y）。

[[Image:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|A misleading

[图片: 熵-互信息-相对熵-关系图. svg | thumb | 256px | 右 | 误导

In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.

[math]\displaystyle{ \Eta(X,Y) \leq \Eta(X) + \Eta(Y) }[/math]

(数学) Eta (x，y) leq Eta (x) + Eta (y)

Definition

[math]\displaystyle{ \Eta(X_1,\ldots, X_n) \leq \Eta(X_1) + \ldots + \Eta(X_n) }[/math]

[ math ] Eta (x _ 1，ldots，x _ n) leq Eta (x _ 1) + ldots + Eta (x _ n)

The joint Shannon entropy (in bits) of two discrete random variables [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] with images [math]\displaystyle{ \mathcal X }[/math] and [math]\displaystyle{ \mathcal Y }[/math] is defined as

[math]\displaystyle{ \Eta(X,Y) = -\sum_{x\in\mathcal X} \sum_{y\in\mathcal Y} P(x,y) \log_2[P(x,y)] }[/math]

(Eq.1)

[math]\displaystyle{ \Eta(X|Y) = \Eta(X,Y) - \Eta(Y)\, }[/math],

埃塔(x

[math]\displaystyle{ \operatorname{I}(X;Y) = \Eta(X) + \Eta(Y) - \Eta(X,Y)\, }[/math]

(x; y) = Eta (x) + Eta (y)-Eta (x，y) ，</math >

where [math]\displaystyle{ x }[/math] and [math]\displaystyle{ y }[/math] are particular values of [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math], respectively, [math]\displaystyle{ P(x,y) }[/math] is the joint probability of these values occurring together, and [math]\displaystyle{ P(x,y) \log_2[P(x,y)] }[/math] is defined to be 0 if [math]\displaystyle{ P(x,y)=0 }[/math].

In quantum information theory, the joint entropy is generalized into the joint quantum entropy.

在量子信息论中，联合熵被推广到联合量子熵。

For more than two random variables [math]\displaystyle{ X_1, ..., X_n }[/math] this expands to

{{Equation box 1

A python package for computing all multivariate joint entropies, mutual informations, conditional mutual information, total correlations, information distance in a dataset of n variables is available.

计算所有多元联合熵，互信息，条件互信息，总相关性，信息距离在一个 n 个变量的数据集的 python 包是可用的。

|indent =

|title=

{{Equation box 1

{方程式方框1

|equation =

[math]\displaystyle{ \Eta(X_1, ..., X_n) = |indent = 2012年10月22日 -\sum_{x_1 \in\mathcal X_1} ... \sum_{x_n \in\mathcal X_n} P(x_1, ..., x_n) \log_2[P(x_1, ..., x_n)] }[/math]

(Eq.2)

|title=

2012年10月11日

|cellpadding= 6

|equation = }}

| equation = }

|border

|cellpadding= 6

6

|border colour = #0073CF

|border

边界

|background colour=#F5FFFA}}

|border colour = #0073CF

0073CF

|background colour=#F5FFFA}}

5/fffa }}

where [math]\displaystyle{ x_1,...,x_n }[/math] are particular values of [math]\displaystyle{ X_1,...,X_n }[/math], respectively, [math]\displaystyle{ P(x_1, ..., x_n) }[/math] is the probability of these values occurring together, and [math]\displaystyle{ P(x_1, ..., x_n) \log_2[P(x_1, ..., x_n)] }[/math] is defined to be 0 if [math]\displaystyle{ P(x_1, ..., x_n)=0 }[/math].

For more than two continuous random variables [math]\displaystyle{ X_1, ..., X_n }[/math] the definition is generalized to:

对于两个以上的连续随机变量 < math > x _ 1，... ，x _ n </math > ，定义被推广到:

Properties

{{Equation box 1

{方程式方框1

Nonnegativity

|indent =

2012年10月22日

|title=

2012年10月11日

The joint entropy of a set of random variables is a nonnegative number.

|equation = }}

| equation = }

|cellpadding= 6

6

[math]\displaystyle{ \Eta(X,Y) \geq 0 }[/math]

|border

边界

|border colour = #0073CF

0073CF

[math]\displaystyle{ \Eta(X_1,\ldots, X_n) \geq 0 }[/math]

|background colour=#F5FFFA}}

5/fffa }}

Greater than individual entropies

The integral is taken over the support of [math]\displaystyle{ f }[/math]. It is possible that the integral does not exist in which case we say that the differential entropy is not defined.

这个积分取代了“数学”的支持。这个积分可能不存在，在这种情况下，我们说微分熵是没有定义的。

The joint entropy of a set of variables is greater than or equal to the maximum of all of the individual entropies of the variables in the set.

As in the discrete case the joint differential entropy of a set of random variables is smaller or equal than the sum of the entropies of the individual random variables:

在离散情况下，一组随机变量的联合微分熵小于或等于各个随机变量的熵之和:

[math]\displaystyle{ \Eta(X,Y) \geq \max \left[\Eta(X),\Eta(Y) \right] }[/math]

[math]\displaystyle{ h(X_1,X_2, \ldots,X_n) \le \sum_{i=1}^n h(X_i) }[/math]

[ math ] h (x _ 1，x _ 2，ldots，x _ n) le sum { i = 1} ^ n h (xi) </math >

[math]\displaystyle{ \Eta \bigl(X_1,\ldots, X_n \bigr) \geq \max_{1 \le i \le n} The following chain rule holds for two random variables: 下面的链式规则适用于两个随机变量: \Bigl\{ \Eta\bigl(X_i\bigr) \Bigr\} }[/math]

[math]\displaystyle{ h(X,Y) = h(X|Y) + h(Y) }[/math]

H (x，y) = h (x | y) + h (y) </math >

In the case of more than two random variables this generalizes to:

对于两个以上的随机变量，这种情况可以推广到:

Less than or equal to the sum of individual entropies

[math]\displaystyle{ h(X_1,X_2, \ldots,X_n) = \sum_{i=1}^n h(X_i|X_1,X_2, \ldots,X_{i-1}) }[/math]

< math > h (x _ 1，x _ 2，ldots，x _ n) = sum _ { i = 1} ^ n h (x _ i | x _ 1，x _ 2，ldots，x _ { i-1}) </math >

Joint differential entropy is also used in the definition of the mutual information between continuous random variables:

联合微分熵也用于连续随机变量之间互信息的定义:

The joint entropy of a set of variables is less than or equal to the sum of the individual entropies of the variables in the set. This is an example of subadditivity. This inequality is an equality if and only if [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] are statistically independent.

[math]\displaystyle{ \operatorname{I}(X,Y)=h(X)+h(Y)-h(X,Y) }[/math]

(x，y) = h (x) + h (y)-h (x，y) </math >

[math]\displaystyle{ \Eta(X,Y) \leq \Eta(X) + \Eta(Y) }[/math]

[math]\displaystyle{ \Eta(X_1,\ldots, X_n) \leq \Eta(X_1) + \ldots + \Eta(X_n) }[/math]

Category:Entropy and information

类别: 熵和信息

Relations to other entropy measures

de:Bedingte Entropie#Blockentropie

1. blockentropie

This page was moved from wikipedia:en:Joint entropy. Its edit history can be viewed at 联合熵/edithistory