微分熵

此词条暂由彩云小译翻译，翻译字数共1092，未经人工整理和审校，带来阅读不便，请见谅。

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

微分熵(也称为连续熵)是信息论中的一个概念，最初由香农尝试将(香农)熵的概念扩展到连续的概率分布，香农熵是衡量一个随机变量的平均惊人程度的指标。不幸的是，香农没有推导出这个公式，而只是假设它是离散熵的正确连续模拟，但它不是。离散熵的实际连续形式是离散点的极限密度(LDDP)。微分熵(在这里描述)在文献中经常遇到，但是它是 LDDP 的一个极限情况，并且失去了它与离散熵的基本联系。

Definition

Let [math]\displaystyle{ X }[/math] be a random variable with a probability density function [math]\displaystyle{ f }[/math] whose support is a set [math]\displaystyle{ \mathcal X }[/math]. The differential entropy [math]\displaystyle{ h(X) }[/math] or [math]\displaystyle{ h(f) }[/math] is defined as

Let [math]\displaystyle{ X }[/math] be a random variable with a probability density function [math]\displaystyle{ f }[/math] whose support is a set [math]\displaystyle{ \mathcal X }[/math]. The differential entropy [math]\displaystyle{ h(X) }[/math] or [math]\displaystyle{ h(f) }[/math] is defined as

设 x 是一个随机变量，其概率密度函数是一个集合。微分熵被定义为

{{Equation box 1

{{Equation box 1

{方程式方框1

|indent =

|indent =

2012年10月22日

|title=

|title=

2012年10月11日

|equation = [math]\displaystyle{ h(X) = -\int_\mathcal{X} f(x)\log f(x)\,dx }[/math]

|equation = [math]\displaystyle{ h(X) = -\int_\mathcal{X} f(x)\log f(x)\,dx }[/math]

| 等式 = < math > h (x) =-int _ mathcal { x } f (x) log f (x) ，dx </math >

|cellpadding= 6

|cellpadding= 6

6

|border

|border

边界

|border colour = #0073CF

|border colour = #0073CF

0073CF

|background colour=#F5FFFA}}

|background colour=#F5FFFA}}

5/fffa }}

For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, [math]\displaystyle{ Q(p) }[/math], then [math]\displaystyle{ h(Q) }[/math] can be defined in terms of the derivative of [math]\displaystyle{ Q(p) }[/math] i.e. the quantile density function [math]\displaystyle{ Q'(p) }[/math] as

For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, [math]\displaystyle{ Q(p) }[/math], then [math]\displaystyle{ h(Q) }[/math] can be defined in terms of the derivative of [math]\displaystyle{ Q(p) }[/math] i.e. the quantile density function [math]\displaystyle{ Q'(p) }[/math] as

对于没有明确的密度函数表达式，但是有明确的分位函数表达式的概率分布，那么可以用 < math > q (p) </math > 的导数来定义 < math > q (p) </math > 。分位数密度函数 < math > q’(p) </math >

[math]\displaystyle{ h(Q) = \int_0^1 \log Q'(p)\,dp }[/math].

[math]\displaystyle{ h(Q) = \int_0^1 \log Q'(p)\,dp }[/math].

< math > h (q) = int _ 0 ^ 1 log q’(p) ，dp </math > 。

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure [math]\displaystyle{ X }[/math]. For example, the differential entropy of a quantity measured in millimeters will be 模板:Not a typo more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of 模板:Not a typo more than the same quantity divided by 1000.

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure [math]\displaystyle{ X }[/math]. For example, the differential entropy of a quantity measured in millimeters will be more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of more than the same quantity divided by 1000.

和它的离散类似物一样，微分熵的单位依赖于对数的底，通常是2(也就是说，单位是位)。请参阅对数单位的对数采取在不同的基地。相关的概念，如联合，条件微分熵，和相对熵，都是以类似的方式定义的。与离散模拟不同，微分熵的偏移量取决于用于测量 < math > x </math > 的单位。例如，以毫米为单位测量的量的微分熵将大于以米为单位测量的相同量; 一个无量纲量的微分熵将大于相同量除以1000。

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution [math]\displaystyle{ \mathcal{U}(0,1/2) }[/math] has negative differential entropy

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution [math]\displaystyle{ \mathcal{U}(0,1/2) }[/math] has negative differential entropy

因为概率密度函数可以大于1，所以在尝试将离散熵的性质应用于微分熵时必须小心谨慎。例如，均匀分布 < math > mathcal { u }(0,1/2) </math > 具有负微分熵

[math]\displaystyle{ \int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\, }[/math].

[math]\displaystyle{ \int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\, }[/math].

2 log (2) ，dx =-log (2) ，</math > .

Thus, differential entropy does not share all properties of discrete entropy.

Thus, differential entropy does not share all properties of discrete entropy.

因此，微分熵并不具有离散熵的所有属性。

Note that the continuous mutual information [math]\displaystyle{ I(X;Y) }[/math] has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps), including linear transformations of [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math], and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

|-

|-

! Distribution Name !! Probability density function (pdf) !! Entropy in nats || Support

!发行名称! ！概率密度函数(pdf)！Nats 中的熵 | 支持

For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.

|-

|-

| Uniform || [math]\displaystyle{ f(x) = \frac{1}{b-a} }[/math] || [math]\displaystyle{ \ln(b - a) \, }[/math] ||[math]\displaystyle{ [a,b]\, }[/math]

| 统一 | | < 数学 > f (x) = frac {1}{ b-a } </math > | < math > ln (b-a) ，</math > | < math > [ a，b ] ，</math >

Properties of differential entropy

|-

|-

• For probability densities [math]\displaystyle{ f }[/math] and [math]\displaystyle{ g }[/math], the Kullback–Leibler divergence [math]\displaystyle{ D_{KL}(f || g) }[/math] is greater than or equal to 0 with equality only if [math]\displaystyle{ f=g }[/math] almost everywhere. Similarly, for two random variables [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math], [math]\displaystyle{ I(X;Y) \ge 0 }[/math] and [math]\displaystyle{ h(X|Y) \le h(X) }[/math] with equality if and only if [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] are independent.

| Normal || [math]\displaystyle{ f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) }[/math] || [math]\displaystyle{ \ln\left(\sigma\sqrt{2\,\pi\,e}\right) }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| 正常 | | < math > f (x) = frac {1}{2 pi sigma ^ 2} exp left (- frac {(x-mu) ^ 2}{2 sigma ^ 2} right) </math > | < math > 左(sigma sqrt {2，pi，e } right) </math | < math > </math > (- infty，infty) ，</math >

• The chain rule for differential entropy holds as in the discrete case

|-

|-

[math]\displaystyle{ h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum_{i=1}^{n} h(X_i) }[/math].

| Exponential || [math]\displaystyle{ f(x) = \lambda \exp\left(-\lambda x\right) }[/math] || [math]\displaystyle{ 1 - \ln \lambda \, }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

指数 | | < math > f (x) = lambda exp left (- lambda x right) </math > | < math > 1-ln lambda，</math > | < math > [0，infty ] ，</math >

• Differential entropy is translation invariant, i.e. for a constant [math]\displaystyle{ c }[/math].

|-

|-

[math]\displaystyle{ h(X+c) = h(X) }[/math]

| Rayleigh || [math]\displaystyle{ f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right) }[/math] || [math]\displaystyle{ 1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2} }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Rayleigh | | < math > f (x) = frac { x }{ sigma ^ 2} exp left (- frac { x ^ 2}{2 sigma ^ 2} right) </math > | < math > | < math > 1 + ln frac { sigma }{ sqrt {2}} + frac { e }{2} </math > | < math > [0，infty ] ，</math >

• Differential entropy is in general not invariant under arbitrary invertible maps.

|-

|-

In particular, for a constant [math]\displaystyle{ a }[/math]

| Beta || [math]\displaystyle{ f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)} }[/math] for [math]\displaystyle{ 0 \leq x \leq 1 }[/math] || [math]\displaystyle{ \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\, }[/math]
[math]\displaystyle{ - (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \, }[/math]||[math]\displaystyle{ [0,1]\, }[/math]

| Beta| | < math > f (x) = frac { x ^ { alpha-1}(1-x) ^ { beta-1}{ b (alpha，beta)}{ math > for < math > 0 leq x leq 1 </math > | | < math > ln b (alpha，beta)-(alpha-1)[ psi (alpha)-psi (alpha + beta)] ，</math > < br/> < math >-(beta-1)[ psi (beta)-psi (alpha + beta)] ，</math > | | < math > [0,1] ，</math >

[math]\displaystyle{ h(aX) = h(X)+ \log |a| }[/math]

|-

|-

For a vector valued random variable [math]\displaystyle{ \mathbf{X} }[/math] and an invertible (square) matrix [math]\displaystyle{ \mathbf{A} }[/math]

| Cauchy || [math]\displaystyle{ f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2} }[/math] || [math]\displaystyle{ \ln(4\pi\gamma) \, }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| Cauchy | | < math > f (x) = frac { gamma }{ pi }{ pi ^ 2 + x ^ 2} </math > | < math > ln (4pi gamma) ，</math > | < math > (- infty，infty) ，</math >

[math]\displaystyle{ h(\mathbf{A}\mathbf{X})=h(\mathbf{X})+\log \left( |\det \mathbf{A}| \right) }[/math]

|-

|-

• In general, for a transformation from a random vector to another random vector with same dimension [math]\displaystyle{ \mathbf{Y}=m \left(\mathbf{X}\right) }[/math], the corresponding entropies are related via

| Chi || [math]\displaystyle{ f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right) }[/math] || [math]\displaystyle{ \ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2} }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Chi | | < math > f (x) = frac {2}{2 ^ { k/2} Gamma (k/2)}} x ^ { k-1} exp left (- frac { x ^ 2}{2}{右) </math > | < math > ln { frac {(k/2)}}{2}}}}-frac {2} psi (frac { k }{2}右) + frac {2}{2} </math > | | math > [0，infty) ，</math >

[math]\displaystyle{ h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx }[/math]

|-

|-

where [math]\displaystyle{ \left\vert \frac{\partial m}{\partial x} \right\vert }[/math] is the Jacobian of the transformation [math]\displaystyle{ m }[/math]. The above inequality becomes an equality if the transform is a bijection. Furthermore, when [math]\displaystyle{ m }[/math] is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and [math]\displaystyle{ h(Y)=h(X) }[/math].

| Chi-squared || [math]\displaystyle{ f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right) }[/math] || [math]\displaystyle{ \ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2} }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Chi-squared | < math > f (x) = frac {1}{2 ^ { k/2} Gamma (k/2)} x ^ { frac { k }{2} !-! 1} exp left (- frac { x }{2}右) </math > | < math > | < math > ln 2 Gamma left (frac { k }{2}右)-left (1-frac { k }{2}右)左(frac { k }2}右) + c { k {2}{ infmath | < < math > [0，fraty) ，</math >

• If a random vector [math]\displaystyle{ X \in \mathbb{R}^n }[/math] has mean zero and covariance matrix [math]\displaystyle{ K }[/math], [math]\displaystyle{ h(\mathbf{X}) \leq \frac{1}{2} \log(\det{2 \pi e K}) = \frac{1}{2} \log[(2\pi e)^n \det{K}] }[/math] with equality if and only if [math]\displaystyle{ X }[/math] is jointly gaussian (see below).

|-

|-

| Erlang || [math]\displaystyle{ f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x) }[/math] || [math]\displaystyle{ (1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Erlang | | < math > f (x) = frac { lambda ^ k }{(k-1) ! }X ^ { k-1} exp (- lambda x) </math > | < math > (1-k) psi (k) + ln frac { Gamma (k)}{ lambda } + k </math > | < math > [0，infty ] ，</math >

However, differential entropy does not have other desirable properties:

|-

|-

• It is not invariant under change of variables, and is therefore most useful with dimensionless variables.

| F || [math]\displaystyle{ f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}} }[/math] || [math]\displaystyle{ \ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) - }[/math]
[math]\displaystyle{ \left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right) }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

我们会找到你的| | < math > f (x) = frac{ n _ 1 ^ { frac { n _ 1}{2}{ frac { n _ 2}{2}}{ b (frac { n _ 1}{2} ，frac { n _ 2}{2}}}}}} frac { x ^ { frac { n _ 1}{2}-1}{(n _ 2 + n _ 1 x) ^ { frac { n _ 1 + n _ 2}{2}}{2}{2}} </} </math > | | | (frac { n _ 1}{ n _ 2} b left (frac { n _ 1}{2} ,2}{2}{2}{2}{2}{2}{1}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{1}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2}{2} psi 左(frac { n _ 1！+\！[0，infty) ，</math > | < math >

• It can be negative.

|-

|-

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

| Gamma || [math]\displaystyle{ f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)} }[/math] || [math]\displaystyle{ \ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Gamma | | < math > f (x) = frac { x ^ { k-1} exp (- frac { x }{ theta })}{ theta ^ k Gamma (k)} </math > | < math > ln (theta Gamma (k)) + (1-k) psi (k) + k，</math > | < math > [0，infty) ，</math >

|-

|-

Maximization in the normal distribution

| Laplace || [math]\displaystyle{ f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right) }[/math] || [math]\displaystyle{ 1 + \ln(2b) \, }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| Laplace | | < math > f (x) = frac {1}{2b } exp left (- frac { | x-mu | }{ b } right) </math > | < math > 1 + ln (2b) ，</math > | < math > (- infty，infty) ，</math >

Theorem

|-

|-

With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.

| Logistic || [math]\displaystyle{ f(x) = \frac{e^{-x}}{(1 + e^{-x})^2} }[/math] || [math]\displaystyle{ 2 \, }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| Logistic | | < math > f (x) = frac { e ^ {-x }{(1 + e ^ {-x }) ^ 2} </math > | < math > 2，</math > | < math > (- infty，infty) ，</math >

|-

|-

Proof

| Lognormal || [math]\displaystyle{ f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right) }[/math] || [math]\displaystyle{ \mu + \frac{1}{2} \ln(2\pi e \sigma^2) }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Lognormal | < math > f (x) = frac {1}{ sigma x sqrt {2 pi } exp left (- frac {(ln x-mu) ^ 2}{2 sigma ^ 2} right) </math > | < math > mu + frac {1}{2} ln (2 pi e sigma ^ 2) </math > | < math > [0，infty) ，</math >

Let [math]\displaystyle{ g(x) }[/math] be a Gaussian PDF with mean μ and variance [math]\displaystyle{ \sigma^2 }[/math] and [math]\displaystyle{ f(x) }[/math] an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that [math]\displaystyle{ f(x) }[/math] has the same mean of [math]\displaystyle{ \mu }[/math] as [math]\displaystyle{ g(x) }[/math].

|-

|-

| Maxwell–Boltzmann || [math]\displaystyle{ f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right) }[/math] || [math]\displaystyle{ \ln(a\sqrt{2\pi})+\gamma_E-\frac{1}{2} }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Maxwell-Boltzmann | | < math > f (x) = frac {1}{ a ^ 3}{ frac {2}{ pi } ，x ^ {2} exp left (- frac { x ^ 2}{2a ^ 2}右) </math > | < math > ln (a sqrt {2 pi }) + e-frac {1} </math > | | math < 0，infty) ，</math >

Consider the Kullback–Leibler divergence between the two distributions

|-

|-

[math]\displaystyle{ 0 \leq D_{KL}(f || g) = \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx = -h(f) - \int_{-\infty}^\infty f(x)\log(g(x)) dx. }[/math]

| Generalized normal || [math]\displaystyle{ f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2) }[/math] || [math]\displaystyle{ \ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2} }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| 广义正态| | < math > f (x) = frac{2 beta ^ { frac { alpha }{2}{ Gamma (frac { alpha }{2})} x ^ { alpha-1} exp (- beta x ^ 2) </math > | | < math > ln { frac { Gamma (alpha/2)}{2 beta ^ { frac {1}{2}}}}-frac { alpha-1}{2} psi left (frac { alpha }{2} right) + frac { alpha }{2}}{2} </math > | | < math > (- infty，infty) ，</math >

Now note that

|-

|-

[math]\displaystyle{ \begin{align} | Pareto || \lt math\gt f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}} }[/math] || [math]\displaystyle{ \ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha} }[/math]||[math]\displaystyle{ [x_m,\infty)\, }[/math]

| Pareto | < math > f (x) = frac { alpha x _ m ^ alpha }{ x ^ { alpha + 1}} </math > | < math > ln frac { x _ m }{ alpha } + 1 + frac {1}{ alpha } </math > | < math > [ x _ m，infty ] ，</math >

\int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\

|-

|-

&= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\

| Student's t || [math]\displaystyle{ f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})} }[/math] || [math]\displaystyle{ \frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right) }[/math]||[math]\displaystyle{ (-\infty,\infty)\, }[/math]

| Student’s t | < math > f (x) = frac {(1 + x ^ 2/nu) ^ {-frac { nu + 1}{2}}{{ sqrt { nu } b (frac {1}{2} ，frac { nu }{2})} </math | | | < math > frac { nu! + ! 1}{2}(左(frac { nu! + 1}{2}右) !-! 左(frac { nu! + 1}{2}右) !-! 左(frac { nu }{2右) ! + ! { nu }{ b 左(frac {2，c {2}{2}{右)

&= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\

|-

|-

&= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\

| Triangular || [math]\displaystyle{ f(x) = \begin{cases} | 三角形 | | \lt math \gt f (x) = begin { cases } &= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\ \frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt] Frac {2(x-a)}{(b-a)(c-a)} & mathrm { for } a le x leq c，[4 pt ] &= -h(g) \frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c \lt x \le b, \\[4pt] Frac {2(b-x)}{(b-a)(b-c)} & mathrm { for } c \lt x le b，[4 pt ] \end{align} }[/math]

\end{cases}</math> || [math]\displaystyle{ \frac{1}{2} + \ln \frac{b-a}{2} }[/math]||[math]\displaystyle{ [0,1]\, }[/math]

结束{ cases } </math > | | < math > frac {1}{2} + ln frac { b-a }{2} </math > | < math > [0,1] ，</math >

because the result does not depend on [math]\displaystyle{ f(x) }[/math] other than through the variance. Combining the two results yields

|-

|-

[math]\displaystyle{ h(g) - h(f) \geq 0 \! }[/math]

| Weibull || [math]\displaystyle{ f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right) }[/math] || [math]\displaystyle{ \frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1 }[/math]||[math]\displaystyle{ [0,\infty)\, }[/math]

| Weibull | | < math > f (x) = frac { k }{ lambda ^ k } x ^ { k-1} exp left (- frac { x ^ k }{ lambda ^ k } right) </math > | < math > | < math > frac {(k-1) gamma _ e }{ k } + ln frac { lambda }{ k } + 1 </math > | < math > [0，infty) ，</math >

with equality when [math]\displaystyle{ f(x)=g(x) }[/math] following from the properties of Kullback–Leibler divergence.

|-

|-

| Multivariate normal || [math]\displaystyle{ 多元正态 | | \lt 数学 \gt ===Alternative proof=== f_X(\vec{x}) = }[/math]
[math]\displaystyle{ \frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}} }[/math] || [math]\displaystyle{ \frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\} }[/math]||[math]\displaystyle{ \mathbb{R}^N }[/math]

F _ x (vec { x }) = </math > < br/> < math > frac { exp left (- frac {1}{2}(vec { x }-vec { mu }) ^ top Sigma ^ {-1} cdot (vec { x }-vec { mu }) right)}{(2 pi) ^ { N/2}左 Sigma | right | ^ {1/2} < | < math > | < < | < math > frac {1}{ ln (2 pi e){{ n } | math < | | | > 数学 < bb >

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

|-

|-

|}

|}

[math]\displaystyle{ L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right) }[/math]

Many of the differential entropies are from.

许多熵的差异来自于。

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition [math]\displaystyle{ \left(1=\int_{-\infty}^\infty g(x)\,dx\right) }[/math] and the requirement of fixed variance [math]\displaystyle{ \left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right) }[/math], are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

[math]\displaystyle{ 0=\delta L=\int_{-\infty}^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx }[/math]

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.

如上所述，微分熵并不具有离散熵的所有属性。例如，微分熵可以是负的，也不是连续坐标变换下的不变量。埃德温·汤普森·杰尼斯表明，事实上，上面的表达式不是一个有限的概率集合表达式的正确极限。

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If [math]\displaystyle{ m(x) }[/math] is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

一个修改的微分熵增加了一个不变测度因子来纠正这个错误。如果 < math > m (x) </math > 被进一步限制为概率密度，那么由此产生的概念在信息论中被称为相对熵:

[math]\displaystyle{ g(x)=e^{-\lambda_0-1-\lambda(x-\mu)^2} }[/math]

[math]\displaystyle{ D(p||m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx. }[/math]

< math > d (p | | | m) = int p (x) log frac { p (x)}{ m (x)} ，dx. </math >

Using the constraint equations to solve for λ₀ and λ yields the normal distribution:

The definition of differential entropy above can be obtained by partitioning the range of [math]\displaystyle{ X }[/math] into bins of length [math]\displaystyle{ h }[/math] with associated sample points [math]\displaystyle{ ih }[/math] within the bins, for [math]\displaystyle{ X }[/math] Riemann integrable. This gives a quantized version of [math]\displaystyle{ X }[/math], defined by [math]\displaystyle{ X_h = ih }[/math] if [math]\displaystyle{ ih \le X \le (i+1)h }[/math]. Then the entropy of [math]\displaystyle{ X_h = ih }[/math] is

上面的微分熵的定义可以通过把 < math > x </math > 的范围划分到与样本点 < math > h </math > 相关的箱子里来得到，因为 < math > x </math > Riemann 可积。这给出了一个量化版本的 < math > x </math > ，定义为 < math > x _ h = ih </math > if < math > > ih le x le (i + 1) h </math > 。然后得到了系统的熵

[math]\displaystyle{ g(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} }[/math]

[math]\displaystyle{ H_h=-\sum_i hf(ih)\log (f(ih)) - \sum hf(ih)\log(h). }[/math]

(h)-sum hf (ih) log (f (ih)-sum hf (ih) log (h)

Example: Exponential distribution

The first term on the right approximates the differential entropy, while the second term is approximately [math]\displaystyle{ -\log(h) }[/math]. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be [math]\displaystyle{ \infty }[/math].

右边的第一个术语近似于微分熵，而第二个术语近似于 math >-log (h) </math > 。请注意，这个过程表明，连续随机变量的离散意义上的熵应该是“数学”。

Let [math]\displaystyle{ X }[/math] be an exponentially distributed random variable with parameter [math]\displaystyle{ \lambda }[/math], that is, with probability density function

[math]\displaystyle{ f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0. }[/math]

Its differential entropy is then

Category:Statistical randomness 分类: 统计的随机性 This page was moved from wikipedia:en:Differential entropy. Its edit history can be viewed at 微分熵/edithistory

[math]\displaystyle{ h_e(X)\, }[/math]	[math]\displaystyle{ =-\int_0^\infty \lambda e^{-\lambda x} \log (\lambda e^{-\lambda x})\,dx }[/math]
	[math]\displaystyle{ = -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right) }[/math]
Category:Entropy and information 类别: 熵和信息	[math]\displaystyle{ = -\log \lambda \int_0^\infty f(x)\,dx + \lambda E[X] }[/math] Category:Information theory 范畴: 信息论