更改

In what follows, an expression of the form is considered by convention to be equal to zero whenever . This is justified because <math>\lim_{p \rightarrow 0+} p \log p = 0</math> for any logarithmic base.

In what follows, an expression of the form is considered by convention to be equal to zero whenever . This is justified because <math>\lim_{p \rightarrow 0+} p \log p = 0</math> for any logarithmic base.

下文中，按惯例将 {{math|''p'' = 0}} 时的表达式{{math|''p'' log ''p''}}的值视为等于零，因为<math>\lim_{p \rightarrow 0+} p \log p = 0</math>适用于任何对数底。

下文中，按惯例将 {{math|1=''p'' = 0}} 时的表达式{{math|''p'' log ''p''}}的值视为等于零，因为<math>\lim_{p \rightarrow 0+} p \log p = 0</math>适用于任何对数底。

<math>H = - \sum_{i} p_i \log_2 (p_i)</math>

<math>H = - \sum_{i} p_i \log_2 (p_i)</math>

<math>H = - \sum_{i} p_i \log_2 (p_i)</math>

<math>H = - \sum_{i} p_i \log_2 (p_i)</math>

where  is the probability of occurrence of the -th possible value of the source symbol. This equation gives the entropy in the units of "bits" (per symbol) because it uses a logarithm of base 2, and this base-2 measure of entropy has sometimes been called the shannon in his honor. Entropy is also commonly computed using the natural logarithm (base E (mathematical constant)|, where  is Euler's number), which produces a measurement of entropy in nats per symbol and sometimes simplifies the analysis by avoiding the need to include extra constants in the formulas. Other bases are also possible, but less commonly used. For example, a logarithm of base  will produce a measurement in bytes per symbol, and a logarithm of base 10 will produce a measurement in decimal digits (or hartleys) per symbol.

where  is the probability of occurrence of the -th possible value of the source symbol. This equation gives the entropy in the units of "bits" (per symbol) because it uses a logarithm of base 2, and this base-2 measure of entropy has sometimes been called the shannon in his honor. Entropy is also commonly computed using the natural logarithm (base E (mathematical constant)|, where  is Euler's number), which produces a measurement of entropy in nats per symbol and sometimes simplifies the analysis by avoiding the need to include extra constants in the formulas. Other bases are also possible, but less commonly used. For example, a logarithm of base  will produce a measurement in bytes per symbol, and a logarithm of base 10 will produce a measurement in decimal digits (or hartleys) per symbol.

其中{{math|''p_i''}}是源符号的第{{math|''i''}}个可能值出现的概率。该方程式以比特（每个符号）为单位给出熵，因为它使用以2为底的对数，所以这个熵有时被称为香农熵以表纪念。熵的计算也通常使用自然对数(以[[E (mathematical constant)|{{mvar|e}}]]为底数，其中{{mvar|e}}是欧拉数，其他底数也是可行的，但不常用)，这样就可以测量每个符号的熵值，有时在公式中可以通过避免额外的常量来简化分析。例如以{{nowrap|1=2⁸ = 256}}为底的对数，每个符号将产生以的字节为单位的测量值。以10为底的对数，每个符号将产生以十进制数字(或哈特利)为单位的测量值。

其中{{math|''p_i''}}是源符号的第{{math|''i''}}个可能值出现的概率。该方程以比特（每个符号）为单位给出熵，因为它使用以2为底的对数。为表纪念，这个熵有时被称为香农熵。熵的计算也通常使用自然对数(以[[E (mathematical constant)|{{mvar|e}}]]为底数，其中{{mvar|e}}是欧拉数，其他底数也是可行的，但不常用)，这样就可以测量每个符号的熵值，有时在公式中可以通过避免额外的常量来简化分析。例如以{{nowrap|1=2⁸ = 256}}为底的对数，得出的值就以字节（而非比特）作为单位。以10为底的对数，每个符号将产生以十进制数字(或哈特利)为单位的测量值。

Intuitively, the entropy  of a discrete random variable  is a measure of the amount of uncertainty associated with the value of  when only its distribution is known.

Intuitively, the entropy  of a discrete random variable  is a measure of the amount of uncertainty associated with the value of  when only its distribution is known.

直观的来看，离散型随机变量{{math|''X''}}的熵{{math|''H_X''}}是不确定性度量，当只知道其分布时，它的值与{{math|''X''}}的值相关。

直观的来看，离散型随机变量{{math|''X''}}的熵{{math|''H_X''}}是对不确定性的度量，当只知道其分布时，它的值与{{math|''X''}}的值相关。

 

 

 

The entropy of a source that emits a sequence of  symbols that are independent and identically distributed (iid) is  bits (per message of  symbols). If the source data symbols are identically distributed but not independent, the entropy of a message of length  will be less than .

The entropy of a source that emits a sequence of  symbols that are independent and identically distributed (iid) is  bits (per message of  symbols). If the source data symbols are identically distributed but not independent, the entropy of a message of length  will be less than .

发出{{math|''N''}}个[[独立同分布]] (iid)的符号序列的源，其熵为{{math|''N'' ⋅ ''H''}}位(每个信息{{math|''N''}}符号)。如果源数据符号是同分布但不独立的，则长度为{{math|''N''}}的消息的熵将小于{{math|''N'' ⋅ ''H''}}。

当一个信息源发出了一串含有{{math|''N''}}个符号的序列，且每个符号[[独立同分布]]时，其熵为{{math|''N'' ⋅ ''H''}}位(每个信息{{math|''N''}}符号)。

如果源数据符号是同分布但不独立的，则长度为{{math|''N''}}的消息的熵将小于{{math|''N'' ⋅ ''H''}}。

If one transmits 1000 bits (0s and 1s), and the value of each of these bits is known to the receiver (has a specific value with certainty) ahead of transmission, it is clear that no information is transmitted.  If, however, each bit is independently equally likely to be 0 or 1, 1000 shannons of information (more often called bits) have been transmitted.  Between these two extremes, information can be quantified as follows. If 𝕏 is the set of all messages }} that  could be, and  is the probability of some <math>x \in \mathbb X</math>, then the entropy, , of  is defined:  

If one transmits 1000 bits (0s and 1s), and the value of each of these bits is known to the receiver (has a specific value with certainty) ahead of transmission, it is clear that no information is transmitted.  If, however, each bit is independently equally likely to be 0 or 1, 1000 shannons of information (more often called bits) have been transmitted.  Between these two extremes, information can be quantified as follows. If 𝕏 is the set of all messages }} that  could be, and  is the probability of some <math>x \in \mathbb X</math>, then the entropy, , of  is defined:  

如果一个人发送了1000比特（0s和1s），并且接收者在发送之前就已知这些比特中的每一个的值（具有确定性的特定值），显然就不会发送任何信息。但是，如果每个比特独立且等可能的为0或1时，则已经发送了1000香农信息（通常称为：比特）。在这两个极端之间，信息可以按以下方式进行量化。如果𝕏是{{math|''X''}}可能在的所有消息的集合{{math|{{mset|''x''₁, ..., ''x''_''n''}}}}，且{{math|''p''(''x'')}}是<math>x \in \mathbb X</math>的概率，那么熵、{{math|''H''}}和{{math|''H''}}的定义如下: <ref name = Reza>{{cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22}}</ref>

如果一个人发送了1000比特（0s和1s），然而接收者在发送之前就已知这串比特序列中的每一个位的值，显然这个通信过程并没有任何信息（译注：如果你要告诉我一个我已已经直到的消息，那么本次通信没有传递任何信息）。但是，如果消息未知，且每个比特独立且等可能的为0或1时，则本次通信传输了1000香农的信息（通常称为“比特”）。在这两个极端之间，信息可以按以下方式进行量化。如果𝕏是{{math|''X''}}可能在的所有消息的集合{{math|{{mset|''x''₁, ..., ''x''_''n''}}}}，且{{math|''p''(''x'')}}是<math>x \in \mathbb X</math>的概率，那么熵、{{math|''H''}}和{{math|''H''}}的定义如下: <ref name = Reza>{{cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22}}</ref>

 

 

 

 

:<math> H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in \mathbb{X}} p(x) \log p(x).</math>

:<math> H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in \mathbb{X}} p(x) \log p(x).</math>

(其中:{{math|''I''(''x'')}}是[[自信息]]，表示单个信息的熵贡献;{{math|''I''(''x'')}}{{math|𝔼_''X''}}为{{math|''X''}}的期望。)熵的一个特性是，当消息空间中的所有消息都是等概率{{math|1=''p''(''x'') = 1/''n''}}时熵最大; 也就是说，在{{math|1=''H''(''X'') = log ''n''}}这种情况下，熵是最不可预测的。

(其中:{{math|''I''(''x'')}}是[[自信息]]，表示单个信息的熵贡献;{{math|''I''(''x'')}}{{math|𝔼_''X''}}为{{math|''X''}}的期望。)熵的一个特性是，当消息空间中的所有消息都是等概率{{math|1=''p''(''x'') = 1/''n''}}时熵最大; 也就是说，在{{math|1=''H''(''X'') = log ''n''}}这种情况下，熵是最不可预测的。

The special case of information entropy for a random variable with two outcomes is the binary entropy function, usually taken to the logarithmic base 2, thus having the shannon (Sh) as unit:

The special case of information entropy for a random variable with two outcomes is the binary entropy function, usually taken to the logarithmic base 2, thus having the shannon (Sh) as unit:

对于具有两个结果的随机变量的信息熵，其特殊情况为二进制熵函数（通常用以为底2对数，因此以香农(Sh)为单位）：

对于只有两种可能取值的随机变量的信息熵，其特殊情况为二值熵函数（通常用以为底2对数，因此以香农(Sh)为单位）：

 

 

 

 

 

:<math>H_{\mathrm{b}}(p) = - p \log_2 p - (1-p)\log_2 (1-p).</math>

:<math>H_{\mathrm{b}}(p) = - p \log_2 p - (1-p)\log_2 (1-p).</math>

信息论中其他重要的量包括Rényi熵(一种熵的推广)，微分熵(信息量推广到连续分布)，以及条件互信息。

信息论中其他重要的量包括Rényi熵(一种熵的推广)，微分熵(信息量推广到连续分布)，以及条件互信息。

==Coding theory==

==Coding theory==