7,129
个编辑
更改
跳到导航
跳到搜索
第1行：
第1行： + + + + + − '''Differential entropy''' (also referred to as '''continuous entropy''') is a concept in [[information theory]] that began as an attempt by Shannon to extend the idea of (Shannon) [[information entropy|entropy]], a measure of average [[surprisal]] of a [[random variable]], to continuous [[probability distribution]]s. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not.<ref>{{cite journal |author=Jaynes, E.T. |authorlink=Edwin Thompson Jaynes |title=Information Theory And Statistical Mechanics |journal=Brandeis University Summer Institute Lectures in Theoretical Physics |volume=3 |issue=sect. 4b |year=1963 |url=http://bayes.wustl.edu/etj/articles/brandeis.pdf |format=PDF}}</ref>{{rp|181–218}} The actual continuous version of discrete entropy is the [[limiting density of discrete points]] (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete [[information entropy|entropy]]. − Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not.+ − + − <math>h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum_{i=1}^{n} h(X_i)</math>.+ − + − − ==Definition定义== − − Let <math>X</math> be a random variable with a [[probability density function]] <math>f</math> whose [[support (mathematics)|support]] is a set <math>\mathcal X</math>. The ''differential entropy'' <math>h(X)</math> or <math>h(f)</math> is defined as<ref name="cover_thomas">{{cite book|first1=Thomas M.|first2=Joy A.|last1=Cover|last2=Thomas|isbn=0-471-06259-6|title=Elements of Information Theory|year=1991|publisher=Wiley|location=New York|url=https://archive.org/details/elementsofinform0000cove|url-access=registration}}</ref> − <math>h(X+c) = h(X)</math> − − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】此处缺无格式的英文及翻译 补充：设随机变量X，其概率密度函数F的的定义域是X的集合 − For probability distributions which don't have an explicit density function expression, but have an explicit [[quantile function]] expression, <math>Q(p)</math>, then <math>h(Q)</math> can be defined in terms of the derivative of <math>Q(p)</math> i.e. the quantile density function <math>Q'(p)</math> as <ref>{{Citation |last1=Vasicek |first1=Oldrich |year=1976 |title=A Test for Normality Based on Sample Entropy |journal=[[Journal of the Royal Statistical Society, Series B]] |volume=38 |issue=1 |jstor=2984828 |postscript=. }}</ref>{{rp|54–59}} − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】此处缺无格式的英文及翻译 补充：For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, , then can be defined in terms of the derivative of i.e. the quantile density function as + − 对于没有显式密度函数表达式，但有显式分位数函数表达式的概率分布，我们则可以用分位数密度函数的导数来定义，即 − A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points). − − 针对这些缺点，提出了一个改进的概念，即相对熵，也被称为 Kullback-Leibler 分歧，其中包括一个不变测度因子(见离散点的极限密度)。 + − As with its discrete analog, the units of differential entropy depend on the base of the [[logarithm]], which is usually 2 (i.e., the units are [[bit]]s). See [[logarithmic units]] for logarithms taken in different bases. Related concepts such as [[joint entropy|joint]], [[conditional entropy|conditional]] differential entropy, and [[Kullback–Leibler divergence|relative entropy]] are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure <math>X</math>.<ref name="gibbs">{{cite book |last=Gibbs |first=Josiah Willard |authorlink=Josiah Willard Gibbs |title=[[Elementary Principles in Statistical Mechanics|Elementary Principles in Statistical Mechanics, developed with especial reference to the rational foundation of thermodynamics]] |year=1902 |publisher=Charles Scribner's Sons |location=New York}}</ref>{{rp|183–184}} For example, the differential entropy of a quantity measured in millimeters will be {{not a typo|log(1000)}} more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of {{not a typo|log(1000)}} more than the same quantity divided by 1000.+ − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：与离散模型一样，微分熵的单位取决于对数的底数，通常是2（单位：比特；请参阅对数单位，了解不同基数的对数。）相对熵的定义与联合熵、条件差分熵等概念相对熵的概念存在类似之处。与离散模型不同，差分熵的偏移量取决于测量单位。例如，以毫米为单位测量的量的差分熵将大于以米为单位测量的相同量；无量纲量的差分熵将大于相同量除以1000。 − − One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the [[Uniform distribution (continuous)|uniform distribution]] <math>\mathcal{U}(0,1/2)</math> has ''negative'' differential entropy − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：在尝试将离散熵的性质应用于微分熵时必须小心，因为概率密度函数可以大于1。例如，均匀分布具有“负”微分熵 − Thus, differential entropy does not share all properties of discrete entropy.+ − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：因此，微分熵并不具有离散熵的所有性质。 − Note that the continuous [[mutual information]] <math>I(X;Y)</math> has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of ''partitions'' of <math>X</math> and <math>Y</math> as these partitions become finer and finer. Thus it is invariant under non-linear [[homeomorphisms]] (continuous and uniquely invertible maps), <ref>{{cite journal | first = Alexander | last = Kraskov |author2=Stögbauer, Grassberger | year = 2004 | title = Estimating mutual information | journal = [[Physical Review E]] | volume = 60 | pages = 066138 | doi =10.1103/PhysRevE.69.066138|arxiv = cond-mat/0305641 |bibcode = 2004PhRvE..69f6138K }}</ref> including linear <ref name = Reza>{{ cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22&as_brr=0&ei=zP79Ro7UBovqoQK4g_nCCw&sig=j3lPgyYrC3-bvn1Td42TZgTzj0Q }}</ref> transformations of <math>X</math> and <math>Y</math>, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values. − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：注意，连续相互变量I（X；Y）具有保留其作为离散信息度量的基本意义的区别，因为它实际上是X和Y的“分区”的离散互信息的极限，因为这些分区变得越来越细。因此，它在非线性[[同胚]]（连续且唯一可逆的映射）下是不变的，并且仍然表示可在允许连续值空间的信道上传输的离散信息量。+ − For the direct analogue of discrete entropy extended to the continuous space, see [[limiting density of discrete points]]. − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：对于扩展到连续空间的离散熵的直接模拟，参见[[离散点的极限密度]]。+ − ==Properties of differential entropy== − 微分熵的性质 − * For probability densities <math>f</math> and <math>g</math>, the [[Kullback–Leibler divergence]] <math>D_{KL}(f || g)</math> is greater than or equal to 0 with equality only if <math>f=g</math> [[almost everywhere]]. Similarly, for two random variables <math>X</math> and <math>Y</math>, <math>I(X;Y) \ge 0</math> and <math>h(X|Y) \le h(X)</math> with equality [[if and only if]] <math>X</math> and <math>Y</math> are [[Statistical independence|independent]]. − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：*对于概率密度f和g，[[Kullback–Leibler散度]]D{KL}（f | | g）</math>只有在f=g[[几乎处处]]时才大于或等于0且相等。类似地，对于两个随机变量X和Y，I（X；Y）\ge 和h（X | Y）\le h（X），等式：当且仅当>X和Y是[[统计独立性|独立性]]。+ − + − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：微分熵的链式法则在离散情况下成立+ − + − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译:微分熵是平移不变的，即对于常数c存在 − + − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：在任意可逆映射下，微分熵一般是不不变的。 − + − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：特别地，对于一个常数a存在 − + − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：对于向量值随机变量X和可逆（平方）矩阵存在+ − :::<math>h(\mathbf{A}\mathbf{X})=h(\mathbf{X})+\log \left( |\det \mathbf{A}| \right)</math><ref name="cover_thomas" />{{rp|253}}+ − * In general, for a transformation from a random vector to another random vector with same dimension <math>\mathbf{Y}=m \left(\mathbf{X}\right)</math>, the corresponding entropies are related via+ − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：一般地，对于从一个随机向量到另一个具有相同维数（X,Y）的随机向量的变换，相应的熵通过+ − − ::<math>h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx</math> − :where <math>\left\vert \frac{\partial m}{\partial x} \right\vert</math> is the [[Jacobian matrix and determinant|Jacobian]] of the transformation <math>m</math>.<ref>{{cite web |title=proof of upper bound on differential entropy of f(X) |work=[[Stack Exchange]] |date=April 16, 2016 |url=https://math.stackexchange.com/q/1745670 }}</ref> The above inequality becomes an equality if the transform is a bijection. Furthermore, when <math>m</math> is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and <math>h(Y)=h(X)</math>.+ − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：其中(m,x)是变换m的[[Jacobian矩阵和行列式| Jacobian]]。如果变换是双射，则上述不等式变为等式。此外，当m是刚性旋转、平移或其组合时，雅可比行列式总是1，并且h（Y）=h（X）+ + + − * If a random vector <math>X \in \mathbb{R}^n</math> has mean zero and [[covariance]] matrix <math>K</math>, <math>h(\mathbf{X}) \leq \frac{1}{2} \log(\det{2 \pi e K}) = \frac{1}{2} \log[(2\pi e)^n \det{K}]</math> with equality if and only if <math>X</math> is [[Multivariate normal distribution#Joint normality|jointly gaussian]] (see [[#Maximization in the normal distribution|below]]).<ref name="cover_thomas" />{{rp|254}} − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：如果一个随机向量X具有均值零和协方差矩阵<math>K</math>，<math>h（\mathbf{X}）\leq\frac{1}{2}\log（\det{2\pi e K}）=\frac{1}{2}\log[（2\pi e）^n\det{K}]</math>等式当且仅当X为多元正态分布/联合正态性/联合高斯（见下文[[#正态分布中的最大化]）。+ − * It is not invariant under [[change of variables]], and is therefore most useful with dimensionless variables.+ − 它在变量变化下不是不变的，因此对无量纲变量最有用 + − * It can be negative. − 它可以为负 − A modification of differential entropy that addresses these drawbacks is the '''relative information entropy''', also known as the Kullback–Leibler divergence, which includes an [[invariant measure]] factor (see [[limiting density of discrete points]]).+ − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译： − 解决这些缺点的微分熵的一种改进是“相对信息熵”，也称为Kullback–Leibler散度，它包括一个“不变测度”因子（参见：离散点的极限密度）。 − + − 正态分布中的最大化+ − ===Theorem=== − 理论 − Its differential entropy is then − 它的微分熵就会 − With a [[normal distribution]], differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.<ref name="cover_thomas" /> − 对于正态分布，对于给定的方差，微分熵是最大的。在所有等方差随机变量中，高斯随机变量的熵最大，或者在均值和方差约束下的最大熵分布是高斯分布 + − + − 证明 − Let <math>g(x)</math> be a [[Normal distribution|Gaussian]] [[Probability density function|PDF]] with mean μ and variance <math>\sigma^2</math> and <math>f(x)</math> an arbitrary [[Probability density function|PDF]] with the same variance. Since differential entropy is translation invariant we can assume that <math>f(x)</math> has the same mean of <math>\mu</math> as <math>g(x)</math>. − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：设<math>g（x）</math>是一个[[正态分布|高斯]][[概率密度函数| PDF]]，具有均值μ和方差<math>\sigma^2</math>和<math>f（x）</math>具有相同方差的任意[[概率密度函数| PDF]]。由于微分熵是平移不变性的，我们可以假设<math>f（x）</math>与<math>g（x）</math>具有相同的<math>\mu</math>平均值。+ − Consider the [[Kullback–Leibler divergence]] between the two distributions − :<math> 0 \leq D_{KL}(f || g) = \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx = -h(f) - \int_{-\infty}^\infty f(x)\log(g(x)) dx.</math> − Now note that 第145行：
第104行： − + 第159行：
第118行： − because the result does not depend on <math>f(x)</math> other than through the variance. Combining the two results yields+ − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：因为结果不依赖于<math>f（x）</math>而不是通过方差。将这两个结果结合起来就得到了 − with equality when <math>f(x)=g(x)</math> following from the properties of Kullback–Leibler divergence.+ − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：当f（x）=g（x）</math>遵循Kullback-Leibler散度的性质时相等。 − + − 替代证明 − This result may also be demonstrated using the [[variational calculus]]. A Lagrangian function with two [[Lagrangian multiplier]]s may be defined as:+ − where ''g(x)'' is some function with mean μ. When the entropy of ''g(x)'' is at a maximum and the constraint equations, which consist of the normalization condition <math>\left(1=\int_{-\infty}^\infty g(x)\,dx\right)</math> and the requirement of fixed variance <math>\left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>, are both satisfied, then a small variation δ''g''(''x'') about ''g(x)'' will produce a variation δ''L'' about ''L'' which is equal to zero:+ − Since this must hold for any small δ''g''(''x''), the term in brackets must be zero, and solving for ''g(x)'' yields:+ + − − Using the constraint equations to solve for λ<sub>0</sub> and λ yields the normal distribution: − − :<math>g(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}</math> − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译： − 这个结果也可以用[[变分演算]]来证明。具有两个[[拉格朗日乘子]]的拉格朗日函数可定义为： − − − − ：<math>L=\int{-\infty}^\infty g（x）\ln（g（x））\，dx-\lambda\u 0\左（1-\int{-\infty}^\infty g（x）\，dx\右）-\lambda\左（\sigma^2-\int{-\infty}^\infty g（x）（x-\mu）^2\，dx\右）</math> − − − − 其中“g（x）”是平均μ的函数。当“g（x）”的熵为最大值时，由归一化条件<math>\ left（1=\int{-\infty}^\infty g（x）\，dx\ right）</math>和固定方差<math>\ left（\sigma^2=\int{-\infty}^\infty g（x）（x-\mu）^2\，dx\ right）</math>组成的约束方程均满足，然后，关于“g（x）”的微小变化δ“g”（“x”）将产生关于“L”的变化δ“L”，其等于零： − − − − ：<math>0=\delta L=\int{-\infty}^\infty\delta g（x）\left（\ln（g（x））+1+\lambda\u 0+\lambda（x-\mu）^2\ right）\，dx</math> − − − − 由于这必须适用于任何小δ“g”（“x”），括号中的项必须为零，求解“g（x）”得到： − − − − ：<math>g（x）=e^{-\lambda\u 0-1-\lambda（x-\mu）^2}</math> − + − ：<math>g（x）=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{（x-\mu）^2}{2\sigma^2}}</math>+ − + − − 例子：指数分布 − Let <math>X</math> be an [[exponential distribution|exponentially distributed]] random variable with parameter <math>\lambda</math>, that is, with probability density function − Its differential entropy is then+ 第240行：
第166行： + − Here, <math>h_e(X)</math> was used rather than <math>h(X)</math> to make it explicit that the logarithm was taken to base ''e'', to simplify the calculation. − − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译： − 设<math>X</math>为[[指数分布|指数分布]]随机变量，参数为<math>\lambda</math>，即概率密度函数 − − − − ：<math>f（x）=\lambda e^{-\lambda x}\mbox{for}x\geq 0.</math> + + − 它的微分熵是 − − {| − − |- − − |<math>h\u e（X）\，</math> − − |<math>=-\int\u 0^\infty\lambda e^{-\lambda x}\log（\lambda e^{-\lambda x}）\，dx</math> − − |- − − | − − |<math>=-\left（\int\u 0^\infty（\log\lambda）\lambda e^{-\lambda x}\，dx+\int\u 0^\infty（-\lambda x）\lambda e^{-\lambda x}\，dx\right）</math> − − |- − − | − − |<math>=-\log\lambda\int\u 0^\infty f（x）\，dx+\lambda E[x]</math> − − |- − − | − − |<math>=-\log\lambda+1\，.</math> − − |} − − − − 这里，使用<math>h（X）</math>而不是<math>h（X）</math>来明确对数取基数“e”，以简化计算。 − − ==Relation to estimator error== − The differential entropy yields a lower bound on the expected squared error of an [[estimator]]. For any random variable <math>X</math> and estimator <math>\widehat{X}</math> the following holds:<ref name="cover_thomas" /> − with equality if and only if <math>X</math> is a Gaussian random variable and <math>\widehat{X}</math> is the mean of <math>X</math>. − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译： − ==与估计器误差的关系==+ − 微分熵给出了[[估计量]]的期望平方误差的下界。对于任何随机变量<math>X</math>和估计器<math>\widehat{X}</math>来说，以下条件成立：<ref name=“cover\u thomas”/> − ：<math>\operatorname{E}[（X-\widehat{X}）^2]\ge\frac{1}{2\pi E}E^{2h（X）}</math>+ − 当且仅当<math>X</math>是高斯随机变量，<math>\widehat{X}</math>是<math>X</math>的平均值。+ − ==Differential entropies for various distributions== − In the table below <math>\Gamma(x) = \int_0^{\infty} e^{-t} t^{x-1} dt</math> is the [[gamma function]], <math>\psi(x) = \frac{d}{dx} \ln\Gamma(x)=\frac{\Gamma'(x)}{\Gamma(x)}</math> is the [[digamma function]], <math>B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}</math> is the [[beta function]], and γ<sub>''E''</sub> is [[Euler-Mascheroni constant|Euler's constant]].<ref>{{cite journal |last1=Park |first1=Sung Y. |last2=Bera |first2=Anil K. |year=2009 |title=Maximum entropy autoregressive conditional heteroskedasticity model |journal=Journal of Econometrics |publisher=Elsevier |url=http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper-masterdownload%5C2009519932327055475115776.pdf |access-date=2011-06-02 |archive-url=https://web.archive.org/web/20160307144515/http://wise.xmu.edu.cn/uploadfiles/paper-masterdownload/2009519932327055475115776.pdf |archive-date=2016-03-07 |url-status=dead }}</ref>{{rp|219–230}} − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − + − Many of the differential entropies are from.<ref name="lazorathie">{{cite journal|author=Lazo, A. and P. Rathie|title=On the entropy of continuous probability distributions|journal=IEEE Transactions on Information Theory|year=1978|volume=24 |issue=1|doi=10.1109/TIT.1978.1055832|pages=120–122}}</ref>{{rp|120–122}} − --[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：+ − + + − 在下表中，dt</math>是[[Gamma function]]，<math>\psi（x）=\frac{d}{dx}\ln\Gamma（x）=\frac{\Gamma'（x）}{\Gamma（x）}</math>是[[digamma function]]，<math>B（p，q）=\frac{\Gamma（p+q）}\Gamma（p+q）{/math>是[[beta function]]，γ<sub>''E'</sub>是[[Euler-Mascheroni常数| Euler常数]]。<ref>{引用期刊| last1=Park | first1=Sung Y.| last2=Bera | first2=Anil K.| year=2009 | title=Maximum熵自回归条件异方差模型|期刊=journal of Econometrics | publisher=Elsevier|网址=http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5上传文件%5Cpaper masterdownload%5C2009519932327055475115776.pdf|访问日期=2011-06-02 |存档url=https://web.archive.org/web/20160307144515/http://智慧.xmu.edu.cn/uploadfiles/paper masterdownload/2009519932327055475115776.pdf |存档日期=2016-03-07 | url状态=dead}}</ref>{rp | 219–230} − {| class=“wikitable”style=“b背景：白色"+ − + − |- − ! 分发名称！！概率密度函数（pdf）！！[[Nat（unit）| Nat]]s | |支持中的熵+ − |- − |[[均匀分布（连续）|均匀]]| |<math>f（x）=\frac{1}{b-a}</math>| |<math>\ln（b-a）\，</math>|<math>[a，b]\，</math>+ − |- − |[[正态分布|正态]]| |<math>f（x）=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left（\frac{（x-\mu）^2}{2\sigma^2}\right）</math>|<math>\ln left（\sigma\sqrt{2\，\pi\，e}\right）</math>|<math>（\infty，\infty）\，</math>+ − |- − |[[指数分布|指数]]| |<math>f（x）=\lambda\exp\left（-\lambda x\right）</math>| |<math>1-\ln\lambda\，</math>| |<math>[0，\infty）\，</math>+ + − |- − |[[Rayleigh distribution | Rayleigh]]| |<math>f（x）=\frac{x}{\sigma^2}\exp\left（-\frac{x^2}{2\sigma^2}\right）</math>|<math>1+\ln\frac{\sigma}{\sqrt{2}+\frac{\gamma E}{2}</math>|{math>[0，\infty）\，</math>+ + − |- − |数学>f（x）f（x））=\frac{{x数学> − |- − − |[[Cauchy分布| Cauchy]]| |<math>f（x）=\frac{\gamma}{\pi}\frac{1}{\gamma^2+x^2}</math>|<math>\ln（4\pi\gamma）\，</math>|<math>（-infty，\infty）\，</math> − − |- − − |[中国分布| Chi分布.[中国分布| Chi分布.[中国分布.[中国分布| Chi分布.]].[中国分布|中国分布| | |数学>f（x）x（x）的数学）=\分形{{2{{k/2{k/2}{k/2}γ（k/2）γ（k/2）}}x ^ k-1}x ^ x{2}</math>|</math>[0，\infty）\，</math> − − |- − − |[[卡方分布|卡方]]| |<math>f（x）=\frac{1}{2^{k/2}\Gamma（k/2）}x^{\frac{k}{2}\！-\!1} \exp\left（-\frac{x}{2}\right）</math>{k}{2}\ln 2\Gamma\left（\frac{k}{2}\right）-\left（1-\frac{k}{2}\right）</psi\left（\frac{k}{2}\right）+\frac{k}{2}</math>{k}\infty）\，</math> − − |- − − |[[Erlang分布| Erlang]]| |<math>f（x）=\frac{\lambda^k}{（k-1）！}x^{k-1}\exp（-\lambda x）</math>| |<math>（1-k）\psi（k）+\ln\frac{\Gamma（k）}{\lambda}+k</math>| |<math>[0，\infty）\，</math> − − |- − − |（F）分销部门的分销ӝF]ӝ数学|数学|数学|数学（x）方面的统计{分销部门的分销ӝ分销保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保保n{u 1+n2}{2}}</math>{math>\ln\frac{n{u 1}{n}u 2}B\左（\frac{n{u 1}{2}），\frac{n{u 2}{2}\right）+\left（1-\frac{n{u 1}{2}\right）\psi\ left（\frac{n{u 1}{2}\right）</math><br/><math>\ left（1+\frac{n{u 2}{2}\right）\psi\ left（\frac{n{u 2}\right）+\frac{n{u 1+n{u 2}\psi\ left（\frac{n{u 1}\right）！+\!n\u 2}{2}\右）</math>| |<math>[0，\infty）\，</math> − − |- − − |[[Gamma distribution | Gamma]]| |<math>f（x）=\frac{x^{k-1}\exp（-\frac{x}{\theta}）}{\theta^k\Gamma（k）}</math>|<math>\ln（\theta\Gamma（k））+（1-k）\psi（k）+k\，</math>|<math>[0，\infty）\，</math> − − |- − − |拉普拉斯分布 − − |- − − |[[Logistic distribution | Logistic]]| |<math>f（x）=\frac{e^{-x}{（1+e^{-x}）^2}</math>|<math>2\，</math>|<math>（\infty，\infty）\，</math> − − |- − − |[[Log normal distribution | Lognormal]]| |<math>f（x）=\frac{1}{\sigma x\sqrt{2\pi}}\exp\left（\frac{（\ln x-\mu）^2}{2\sigma^2}\right）</math>| |<math>\mu+\frac{1}{2}\ln 2\pie\sigma^2）</math>|{math>[0，infty）\，</math> − |- − − |[[Maxwell-Boltzmann分布| Maxwell-Boltzmann]]| |<math>f（x）=\frac{1}{a^3}\sqrt{\frac{2}{\pi}}\，x^{2}\exp\左（\frac{x^2}{2a^2}\右）</math>| |<math>\ln（a\sqrt{2\pi}）+\gamma u E-\frac{1}2}</math |<math>[0，infty）\，</math>> − − |- − − |[广义高斯分布[广义高斯分布[广义高斯分布[广义高斯分布[广义高斯分布[广义高斯分布[广义高斯分布[广义高斯分布{\alpha}{2}\右）+\frac{\alpha}{2}</math>| |<math>（-infty，\infty）\，</math> − − |- − − |[[Pareto分布| Pareto]]| |<math>f（x）=\frac{\alpha x{m^\alpha}{x^{\alpha+1}</math>|{math>\ln\frac{x{m}{\alpha}+1+\frac{1}{\alpha}</math>|{math>[x{m，\infty）\，</math> − − |- − − |[[Student's t-distribution | Student's t]]| |<math>f（x）=\frac{（1+x^2/\nu）^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B（\frac{1}{2}，\frac{\nu 2}）}</math>{！+\!1} {2}\左（\psi\左（\frac{\nu\）！+\!1} {2}\对）\！-\!\psi\左（\frac{\nu}{2}\右）\right）\！+\!\ln\sqrt{\nu}B\左（\frac{1}{2}，\frac{\nu}{2}\右）</math>|</math>（\infty，\infty）\，</math> − − |- − − |[[三角分布|三角]]| |<math>f（x）=\begin{cases} − − \frac{2（x-a）}{（b-a）（c-a）}&\mathrm{for\}a\le x\leq c，\\[4pt] − − \frac{2（b-x）}{（b-a）（b-c）}&\mathrm{for\}c<x\le b，\\[4pt] − − \结束{cases}</math>|{math>\frac{1}{2}+\ln\frac{b-a}{2}</math>|{math>[0,1]\，</math> − − |- − − |[[Weibull分布| Weibull]]| |<math>f（x）=\frac{k}{\lambda^k}x^{k-1}\exp\左（\frac{x^k}{\lambda^k}\右）</math>|{math>\frac{（k-1）\gamma E}{k}+\ln frac{\lambda}{k}+1</math>{math>[0，\infty）\，</math> − − |- − − |[[多元正态分布|多元正态]]| |<数学> − − 该公司（vec{{X}）的X（vec{{X}）的X（vec{X{{vec{{X{{X{{{{{{{{{{{{{}}{{{{{}{}}{{e）^{N}\det（\Sigma）\}</math>| |<math>\mathbb{R}^N</math> − − |- − − |} − − − − 许多微分熵来自.<ref name=“lazorathie”>{引用期刊| author=Lazo，A.和P.Rathie | title=关于连续概率分布熵| journal=IEEE Transactions On Information Theory | year=1978 | volume=24 | issue=1 | doi=10.1109/TIT.1978.1055832 | pages=120–122}</ref>{rp | 120–122} −
无编辑摘要
此词条暂由Henry翻译。
此词条暂由Henry翻译。
由CecileLi初步审校。
由CecileLi初步审校。
{{#seo:
|keywords=连续熵,微分熵,信息论
|description=是信息论中的一个概念，其来源于香农尝试将他的香农熵的概念扩展到连续的概率分布
}}
'''微分熵 Differential entropy'''（也被称为'''连续熵 continuous entropy'''）是信息论中的一个概念，其来源于香农尝试将他的香农熵的概念扩展到连续的概率分布。香农熵是衡量一个随机变量的平均惊异程度的指标。可惜的是，香农只是假设它是离散熵的正确连续模拟而并没有推导出公式，但事实上它并不是离散熵的正确连续模拟。<ref>{{cite journal |author=Jaynes, E.T. |authorlink=Edwin Thompson Jaynes |title=Information Theory And Statistical Mechanics |journal=Brandeis University Summer Institute Lectures in Theoretical Physics |volume=3 |issue=sect. 4b |year=1963 |url=http://bayes.wustl.edu/etj/articles/brandeis.pdf |format=PDF}}</ref>离散熵的实际连续版本是离散点的极限密度 limiting density of discrete points（LDDP）。微分熵（此处描述）在文献中很常见，但它是 LDDP 的一个极限情况，并且失去了与离散熵的基本联系。
==定义==
<font color="#ff8000"> 微分熵Differential entropy</font>(也被称为连续熵)是信息论中的一个概念，其来源于香农尝试将他的香农熵的概念扩展到连续的概率分布。香农熵是衡量一个随机变量的平均惊异程度的指标。可惜的是，香农只是假设它是离散熵的正确连续模拟而并没有推导出公式，但事实上它并不是离散熵的正确连续模拟。
Let <math>X</math> be a random variable with a [[probability density function]] <math>f</math> whose [[support (mathematics)|support]] is a set <math>\mathcal X</math>. The ''differential entropy'' <math>h(X)</math> or <math>h(f)</math> is defined as
设随机变量<math>X</math>，其概率密度函数<math>f</math>的的定义域是<math>\mathcal X</math>的集合。该微分熵 <math>h(X)</math> 或者<math>h(f)</math>定义为
<ref name="cover_thomas">{{cite book|first1=Thomas M.|first2=Joy A.|last1=Cover|last2=Thomas|isbn=0-471-06259-6|title=Elements of Information Theory|year=1991|publisher=Wiley|location=New York|url=https://archive.org/details/elementsofinform0000cove|url-access=registration}}</ref>
:<math>h(X) = -\int_\mathcal{X} f(x)\log f(x)\,dx</math>
:<math>h(X) = -\int_\mathcal{X} f(x)\log f(x)\,dx</math>
对于没有显式密度函数表达式，但有显式分位数函数表达式的概率分布，<math>Q(p)</math>，则<math>h(Q)</math>可以用导数<math>Q(p)</math>来定义，即分位数密度函数<math>Q'(p)</math><ref>{{Citation |last1=Vasicek |first1=Oldrich |year=1976 |title=A Test for Normality Based on Sample Entropy |journal=[[Journal of the Royal Statistical Society, Series B]] |volume=38 |issue=1 |jstor=2984828 |postscript=. }}</ref>
:<math>h(Q) = \int_0^1 \log Q'(p)\,dp</math>.
:<math>h(Q) = \int_0^1 \log Q'(p)\,dp</math>.
与离散模型一样，微分熵的单位取决于对数的底数，通常是2（单位：比特；请参阅对数单位，了解不同基数的对数。）相关概念，如[[联合熵]]、[[条件微分熵]]和[[相对熵]]，以类似的方式定义。与离散模型不同，微分熵的偏移量取决于测量单位。<ref name="gibbs">{{cite book |last=Gibbs |first=Josiah Willard |authorlink=Josiah Willard Gibbs |title=[[Elementary Principles in Statistical Mechanics|Elementary Principles in Statistical Mechanics, developed with especial reference to the rational foundation of thermodynamics]] |year=1902 |publisher=Charles Scribner's Sons |location=New York}}</ref>例如，以毫米为单位的量的微分熵将比以米为单位测量的相同量的微分熵大 log(1000)；无量纲量的log(1000)微分熵将大于相同量除以1000。
在尝试将离散熵的性质应用于微分熵时必须小心，因为概率密度函数可以大于1。例如，均匀分布<math>\mathcal{U}(0,1/2)</math>具有“负”微分熵
:<math>\int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\,</math>.
:<math>\int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\,</math>.
因此，微分熵并不具有离散熵的所有性质。
注意，连续[[互信息]]<math>I(X;Y)</math> 具有保留其作为离散信息度量的基本意义的区别，因为它实际上是X和Y的“分区”的离散互信息的极限，因为这些分区变得越来越细。因此，它在非线性同胚（连续且唯一可逆的映射）下是不变的，<ref>{{cite journal | first = Alexander | last = Kraskov |author2=Stögbauer, Grassberger | year = 2004 | title = Estimating mutual information | journal = [[Physical Review E]] | volume = 60 | pages = 066138 | doi =10.1103/PhysRevE.69.066138|arxiv = cond-mat/0305641 |bibcode = 2004PhRvE..69f6138K }}</ref>包括线性<ref name = Reza>{{ cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22&as_brr=0&ei=zP79Ro7UBovqoQK4g_nCCw&sig=j3lPgyYrC3-bvn1Td42TZgTzj0Q }}</ref>变换<math>X</math>和<math>Y</math>，并且仍然表示可在允许连续值空间的信道上传输的离散信息量。
对于扩展到连续空间的离散熵的直接模拟，参见[[离散点的极限密度]]。
==微分熵的性质==
* The chain rule for differential entropy holds as in the discrete case<ref name="cover_thomas" />{{rp|253}}
*对于概率密度<math>f</math>和<math>g</math>，仅当<math>f=g</math>几乎处处成立时,[[Kullback–Leibler散度]]<math>D_{KL}(f || g)</math>才大于或等于0。类似地，对于两个随机变量<math>X</math>和<math>Y</math>，当且仅当<math>X</math>和<math>Y</math>是独立，<math>I(X;Y) \ge 0</math>才和<math>h(X|Y) \le h(X)</math>相等。
* 微分熵的链式法则在离散情况下成立<ref name="cover_thomas" />
::<math>h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum_{i=1}^{n} h(X_i)</math>.
::<math>h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum_{i=1}^{n} h(X_i)</math>.
* Differential entropy is translation invariant, i.e. for a constant <math>c</math>.<ref name="cover_thomas" />{{rp|253}}
* 微分熵是平移不变的，即对于常数<math>c</math>存在。<ref name="cover_thomas" />
::<math>h(X+c) = h(X)</math>
::<math>h(X+c) = h(X)</math>
* Differential entropy is in general not invariant under arbitrary invertible maps.
* 在任意可逆映射下，微分熵通常不是不变的。
:: In particular, for a constant <math>a</math>
::特别地，对于一个常数a，
:::<math>h(aX) = h(X)+ \log |a|</math>
:::<math>h(aX) = h(X)+ \log |a|</math>
:: For a vector valued random variable <math>\mathbf{X}</math> and an invertible (square) [[matrix (mathematics)|matrix]] <math>\mathbf{A}</math>
::对于向量值随机变量<math>\mathbf{X}</math>和可逆（平方）矩阵<math>\mathbf{A}</math>
:::<math>h(\mathbf{A}\mathbf{X})=h(\mathbf{X})+\log \left( |\det \mathbf{A}| \right)</math><ref name="cover_thomas" />
* 一般地，对于从一个随机向量到另一个具有相同维数的随机向量的变换<math>\mathbf{Y}=m \left(\mathbf{X}\right)</math>，相应的熵通过
::<math>h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx</math>
:其中<math>\left\vert \frac{\partial m}{\partial x} \right\vert</math>是变换的[[Jacobian矩阵和行列式| Jacobian]]<math>m</math>。<ref>{{cite web |title=proof of upper bound on differential entropy of f(X) |work=[[Stack Exchange]] |date=April 16, 2016 |url=https://math.stackexchange.com/q/1745670 }}</ref>如果变换是双射，则上述不等式变为等式。此外，当<math>m</math>是刚性旋转、平移或其组合时，雅可比行列式总是1，并且<math>h(Y)=h(X)</math>。
* 如果一个随机向量X具有均值零和协方差矩阵<math>K</math>, <math>h(\mathbf{X}) \leq \frac{1}{2} \log(\det{2 \pi e K}) = \frac{1}{2} \log[(2\pi e)^n \det{K}]</math>相等当且仅当<math>X</math>为多元正态分布/联合正态性/联合高斯（见下文[[#正态分布中的最大化]）。<ref name="cover_thomas" />
然而，微分熵没有其他理想的特性：
* 它在变量变化下不是不变的，因此对无量纲变量最有用。
* 它可以为负。
解决这些缺点的微分熵的一种改进是“相对信息熵”，也称为[[Kullback–Leibler散度]]，它包括一个不变的测度因子（参见：[[离散点的极限密度]]）。
==正态分布中的最大化==
===定理===
对于正态分布，对于给定的方差，微分熵是最大的。在所有等方差随机变量中，高斯随机变量的熵最大，或者在均值和方差约束下的最大熵分布是高斯分布。<ref name="cover_thomas" />
==Maximization in the normal distribution==
===证明===
设<math>g（x）</math>是一个正态分布的概率密度函数，具有均值μ和方差<math>\sigma^2</math>和<math>f（x）</math>具有相同方差的任意概率密度函数。由于微分熵是平移不变性的，我们可以假设<math>f（x）</math>具有相同的均值<math>\mu</math>作为<math>g（x）</math>。
考虑两个分布之间的[[Kullback–Leibler散度]]
===Proof===
:<math> 0 \leq D_{KL}(f || g) = \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx = -h(f) - \int_{-\infty}^\infty f(x)\log(g(x)) dx.</math>
现在请注意
:<math>\begin{align}
:<math>\begin{align}
\int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\
\int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\
\end{align}</math>
\end{align}</math>
--[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：考虑两个分布之间的[[Kullback–Leibler散度]]
--[[用户:CecileLi|CecileLi]]([[用户讨论:CecileLi|讨论]]) 【审校】补充翻译：
：<math>0\leq D{KL}（f{g）=\int{-\infty}^\infty f（x）\log\left（\frac{f（x）}{g（x）}\right）dx=-h（f）-\int{-\infty}^\infty f（x）\log（g（x））dx。</math>
：<math>0\leq D{KL}（f{g）=\int{-\infty}^\infty f（x）\log\left（\frac{f（x）}{g（x）}\right）dx=-h（f）-\int{-\infty}^\infty f（x）\log（g（x））dx。</math>
\end{align}</math>
\end{align}</math>
因为结果不依赖于<math>f(x)</math>而不是通过方差。将这两个结果结合起来就得到了
:<math> h(g) - h(f) \geq 0 \!</math>
:<math> h(g) - h(f) \geq 0 \!</math>
当<math>f(x)=g(x)</math>遵循Kullback-Leibler散度的性质时相等。
===Alternative proof===
===替代证明===
这个结果也可以用[[变分演算]]来证明。具有两个[[拉格朗日乘子]]的拉格朗日函数可定义为：
:<math>L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>
:<math>L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>
其中''g(x)''是平均μ的函数。当''g(x)''的熵为最大值时，由归一化条件<math>\ left（1=\int{-\infty}^\infty g（x）\，dx\ right）</math>和固定方差<math>\left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)</math>组成的约束方程均满足，然后，关于''g(x)''的微小变化δ''g''(''x'')将产生关于''L''的变化δ''L''，其等于零：
:<math>0=\delta L=\int_{-\infty}^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx</math>
:<math>0=\delta L=\int_{-\infty}^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx</math>
由于这必须适用于任何小δ''g''(''x'')，括号中的项必须为零，求解''g(x)''得到：
:<math>g(x)=e^{-\lambda_0-1-\lambda(x-\mu)^2}</math>
:<math>g(x)=e^{-\lambda_0-1-\lambda(x-\mu)^2}</math>
使用约束方程求解λ<sub>0</sub>和λ得出正态分布：
使用约束方程求解λ<sub>0</sub>和λ得出正态分布：
:<math>g(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}</math>
==例子：指数分布==
设<math>X</math>为指数分布随机变量，参数为<math>\lambda</math>，即概率密度函数
==Example: Exponential distribution==
:<math>f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.</math>
:<math>f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.</math>
它的微分熵是
{|
{|
|-
|-
| <math>= -\log\lambda + 1\,.</math>
| <math>= -\log\lambda + 1\,.</math>
|}
|}
这里，<math>h_e(X)</math>被使用而不是<math>h(X)</math>明确以''e''为底对数，以简化计算。
==与估计器误差的关系==
微分熵给出了[[估计量]]的期望平方误差的下界。对于任何随机变量<math>X</math>和估计器<math>\widehat{X}</math>来说，以下条件成立：<ref name=“cover\u thomas”/>
:<math>\operatorname{E}[(X - \widehat{X})^2] \ge \frac{1}{2\pi e}e^{2h(X)}</math>
:<math>\operatorname{E}[(X - \widehat{X})^2] \ge \frac{1}{2\pi e}e^{2h(X)}</math>
当且仅当<math>X</math>是高斯随机变量，<math>\widehat{X}</math>是<math>X</math>的平均值。
==各种分布的微分熵==
在下表中，<math>\Gamma(x) = \int_0^{\infty} e^{-t} t^{x-1} dt</math>是Gamma函数，<math>\psi(x) = \frac{d}{dx} \ln\Gamma(x)=\frac{\Gamma'(x)}{\Gamma(x)}</math>是digamma 函数，<math>B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}</math>是beta函数，γ<sub>''E'</sub>是欧拉常数。<ref>{{cite journal |last1=Park |first1=Sung Y. |last2=Bera |first2=Anil K. |year=2009 |title=Maximum entropy autoregressive conditional heteroskedasticity model |journal=Journal of Econometrics |publisher=Elsevier |url=http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper-masterdownload%5C2009519932327055475115776.pdf |access-date=2011-06-02 |archive-url=https://web.archive.org/web/20160307144515/http://wise.xmu.edu.cn/uploadfiles/paper-masterdownload/2009519932327055475115776.pdf |archive-date=2016-03-07 |url-status=dead }}</ref>
{| class="wikitable" style="background:white"
{| class="wikitable" style="background:white"
|+ Table of differential entropies
|+ Table of differential entropies
|-
|-
! Distribution Name !! Probability density function (pdf) !! Entropy in [[Nat (unit)|nat]]s || Support
! 分布 !! 概率密度函数 !! 信息自然单位中的熵 || 范围
|-
|-
| [[Uniform distribution (continuous)|Uniform]] || <math>f(x) = \frac{1}{b-a}</math> || <math>\ln(b - a) \,</math> ||<math>[a,b]\,</math>
| 连续均匀分布 || <math>f(x) = \frac{1}{b-a}</math> || <math>\ln(b - a) \,</math> ||<math>[a,b]\,</math>
|-
|-
| [[Normal distribution|Normal]] || <math>f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)</math> || <math>\ln\left(\sigma\sqrt{2\,\pi\,e}\right) </math>||<math>(-\infty,\infty)\,</math>
| 正态分布 || <math>f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)</math> || <math>\ln\left(\sigma\sqrt{2\,\pi\,e}\right) </math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Exponential distribution|Exponential]] || <math>f(x) = \lambda \exp\left(-\lambda x\right)</math> || <math>1 - \ln \lambda \, </math>||<math>[0,\infty)\,</math>
| 指数分布 || <math>f(x) = \lambda \exp\left(-\lambda x\right)</math> || <math>1 - \ln \lambda \, </math>||<math>[0,\infty)\,</math>
|-
|-
| [[Rayleigh distribution|Rayleigh]] || <math>f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right)</math> || <math>1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2}</math>||<math>[0,\infty)\,</math>
| 瑞利分布 || <math>f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right)</math> || <math>1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2}</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Beta distribution|Beta]] || <math>f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}</math> for <math>0 \leq x \leq 1</math> || <math> \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,</math><br /><math>- (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \, </math>||<math>[0,1]\,</math>
| Beta分布 || <math>f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}</math> for <math>0 \leq x \leq 1</math> || <math> \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,</math><br /><math>- (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \, </math>||<math>[0,1]\,</math>
|-
|-
| [[Cauchy distribution|Cauchy]] || <math>f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2}</math> || <math>\ln(4\pi\gamma) \, </math>||<math>(-\infty,\infty)\,</math>
| 柯西分布 || <math>f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2}</math> || <math>\ln(4\pi\gamma) \, </math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Chi distribution|Chi]] || <math>f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right)</math> || <math>\ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>[0,\infty)\,</math>
| Chi分布 || <math>f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right)</math> || <math>\ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Chi-squared distribution|Chi-squared]] || <math>f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right)</math> || <math>\ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>[0,\infty)\,</math>
| 卡方分布 || <math>f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right)</math> || <math>\ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2}</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Erlang distribution|Erlang]] || <math>f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x)</math> || <math>(1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k</math>||<math>[0,\infty)\,</math>
| Erlang分布 || <math>f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x)</math> || <math>(1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k</math>||<math>[0,\infty)\,</math>
|-
|-
| [[F distribution|F]] || <math>f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}}</math> || <math>\ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -</math><br /><math>\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right)</math>||<math>[0,\infty)\,</math>
| F分布 || <math>f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}}</math> || <math>\ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -</math><br /><math>\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right)</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Gamma distribution|Gamma]] || <math>f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)}</math> || <math>\ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, </math>||<math>[0,\infty)\,</math>
| Gamma分布|| <math>f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)}</math> || <math>\ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, </math>||<math>[0,\infty)\,</math>
|-
|-
| [[Laplace distribution|Laplace]] || <math>f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right)</math> || <math>1 + \ln(2b) \, </math>||<math>(-\infty,\infty)\,</math>
| 拉普拉斯分布 || <math>f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right)</math> || <math>1 + \ln(2b) \, </math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Logistic distribution|Logistic]] || <math>f(x) = \frac{e^{-x}}{(1 + e^{-x})^2}</math> || <math>2 \, </math>||<math>(-\infty,\infty)\,</math>
| 逻辑分布 || <math>f(x) = \frac{e^{-x}}{(1 + e^{-x})^2}</math> || <math>2 \, </math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Log-normal distribution|Lognormal]] || <math>f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right)</math> || <math>\mu + \frac{1}{2} \ln(2\pi e \sigma^2)</math>||<math>[0,\infty)\,</math>
| 对数正态分布 || <math>f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right)</math> || <math>\mu + \frac{1}{2} \ln(2\pi e \sigma^2)</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Maxwell–Boltzmann distribution|Maxwell–Boltzmann]] || <math>f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right)</math> || <math>\ln(a\sqrt{2\pi})+\gamma_E-\frac{1}{2}</math>||<math>[0,\infty)\,</math>
| 麦克斯韦-玻尔兹曼分布 || <math>f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right)</math> || <math>\ln(a\sqrt{2\pi})+\gamma_E-\frac{1}{2}</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Generalized Gaussian distribution|Generalized normal]] || <math>f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2)</math> || <math>\ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2}</math>||<math>(-\infty,\infty)\,</math>
| 广义正态分布 || <math>f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2)</math> || <math>\ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2}</math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Pareto distribution|Pareto]] || <math>f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}}</math> || <math>\ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha}</math>||<math>[x_m,\infty)\,</math>
| Pareto分布 || <math>f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}}</math> || <math>\ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha}</math>||<math>[x_m,\infty)\,</math>
|-
|-
| [[Student's t-distribution|Student's t]] || <math>f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})}</math> || <math>\frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right)</math>||<math>(-\infty,\infty)\,</math>
| 学生t分布 || <math>f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})}</math> || <math>\frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right)</math>||<math>(-\infty,\infty)\,</math>
|-
|-
| [[Triangular distribution|Triangular]] || <math> f(x) = \begin{cases}
| 三角分布 || <math> f(x) = \begin{cases}
\frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt]
\frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt]
\frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt]
\frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt]
\end{cases}</math> || <math>\frac{1}{2} + \ln \frac{b-a}{2}</math>||<math>[0,1]\,</math>
\end{cases}</math> || <math>\frac{1}{2} + \ln \frac{b-a}{2}</math>||<math>[0,1]\,</math>
|-
|-
| [[Weibull distribution|Weibull]] || <math>f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right)</math> || <math>\frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1</math>||<math>[0,\infty)\,</math>
| 威布尔分布 || <math>f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right)</math> || <math>\frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1</math>||<math>[0,\infty)\,</math>
|-
|-
| [[Multivariate normal distribution|Multivariate normal]] || <math>
| 多元正态分布 || <math>
f_X(\vec{x}) =</math><br /><math> \frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}}</math> || <math>\frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\}</math>||<math>\mathbb{R}^N</math>
f_X(\vec{x}) =</math><br /><math> \frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}}</math> || <math>\frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\}</math>||<math>\mathbb{R}^N</math>
|-
|-
|}
|}
许多微分熵来自.<ref name="lazorathie">{{cite journal|author=Lazo, A. and P. Rathie|title=On the entropy of continuous probability distributions|journal=IEEE Transactions on Information Theory|year=1978|volume=24 |issue=1|doi=10.1109/TIT.1978.1055832|pages=120–122}}</ref>
==各种分布的微分熵==
==变体==
如上所述，微分熵不具有离散熵的所有性质。例如，微分熵可以是负的；在连续坐标变换下也不是不变的。Edwin Thompson Jaynes事实上证明了上面的表达式不是有限概率的表达式的正确限制。<ref>{{cite journal |author=Jaynes, E.T. |author-link=Edwin Thompson Jaynes |title=Information Theory And Statistical Mechanics |journal=Brandeis University Summer Institute Lectures in Theoretical Physics |volume=3 |issue=sect. 4b |year=1963 |url=http://bayes.wustl.edu/etj/articles/brandeis.pdf }}</ref>
微分熵的修改增加了一个不变的度量因子来纠正这个问题，（见离散点的极限密度）。如果<math>m(x)</math>被进一步约束为概率密度，由此产生的概念在信息论中称为'''相对熵 relative entropy'''：
|+微分熵表
:<math>D(p||m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx.</math>
上面的微分熵的定义可以通过划分范围来获得<math>X</math>成箱的长度 {\displaystyle h}H 与相关的样本点<math>h</math>在垃圾箱内，对于<math>X</math>黎曼可积。这给出了一个量化的版本<math>X</math>， 被定义为<math>X_h = ih</math> 如果<math>ih \le X \le (i+1)h</math>. 那么熵<math>X_h = ih</math>是<ref name="cover_thomas"/>
:<math>H_h=-\sum_i hf(ih)\log (f(ih)) - \sum hf(ih)\log(h).</math>
右边的第一项近似于微分熵，而第二项近似于<math>-\log(h)</math>。 请注意，此过程表明连续随机变量的离散意义上的熵应该是<math>\infty</math>。
==参考文献==
<references/>
==编者推荐==
===
---------
---------
[[Category:熵和信息]]
[[Category:熵和信息]]
[[Category:信息论]]
[[Category:信息论]]
[[Category:统计的随机性]]
[[Category:统计的随机性]]