更改

where  is the probability of occurrence of the -th possible value of the source symbol. This equation gives the entropy in the units of "bits" (per symbol) because it uses a logarithm of base 2, and this base-2 measure of entropy has sometimes been called the shannon in his honor. Entropy is also commonly computed using the natural logarithm (base E (mathematical constant)|, where  is Euler's number), which produces a measurement of entropy in nats per symbol and sometimes simplifies the analysis by avoiding the need to include extra constants in the formulas. Other bases are also possible, but less commonly used. For example, a logarithm of base  will produce a measurement in bytes per symbol, and a logarithm of base 10 will produce a measurement in decimal digits (or hartleys) per symbol.

where  is the probability of occurrence of the -th possible value of the source symbol. This equation gives the entropy in the units of "bits" (per symbol) because it uses a logarithm of base 2, and this base-2 measure of entropy has sometimes been called the shannon in his honor. Entropy is also commonly computed using the natural logarithm (base E (mathematical constant)|, where  is Euler's number), which produces a measurement of entropy in nats per symbol and sometimes simplifies the analysis by avoiding the need to include extra constants in the formulas. Other bases are also possible, but less commonly used. For example, a logarithm of base  will produce a measurement in bytes per symbol, and a logarithm of base 10 will produce a measurement in decimal digits (or hartleys) per symbol.

其中{{math|''p_i''}}是源符号的第{{math|''i''}}个可能值出现的概率。该方程以比特（每个符号）为单位给出熵，因为它使用以2为底的对数。为表纪念，这个熵有时被称为香农熵。熵的计算也通常使用自然对数(以[[E (mathematical constant)|{{mvar|e}}]]为底数，其中{{mvar|e}}是欧拉数，其他底数也是可行的，但不常用)，这样就可以测量每个符号的熵值，有时在公式中可以通过避免额外的常量来简化分析。例如以{{nowrap|1=2⁸ = 256}}为底的对数，得出的值就以字节（而非比特）作为单位。以10为底的对数，每个符号将产生以十进制数字(或哈特利)为单位的测量值。

其中{{math|''p_i''}}是源符号的第{{math|''i''}}个可能值出现的概率。该方程以比特（每个符号）为单位给出熵，因为它使用以2为底的对数。为表纪念，这个熵有时被称为香农熵。熵的计算也通常使用自然对数(以[[E (mathematical constant)|{{mvar|e}}]]为底数，其中{{mvar|e}}是欧拉数，其他底数也是可行的，但不常用)，这样就可以测量每个符号的熵值，有时在公式中可以通过避免额外的常量来简化分析。例如以{{math|1=2⁸ = 256}}为底的对数，得出的值就以字节（而非比特）作为单位。以10为底的对数，每个符号将产生以十进制数字(或哈特利)为单位的测量值。

If one transmits 1000 bits (0s and 1s), and the value of each of these bits is known to the receiver (has a specific value with certainty) ahead of transmission, it is clear that no information is transmitted.  If, however, each bit is independently equally likely to be 0 or 1, 1000 shannons of information (more often called bits) have been transmitted.  Between these two extremes, information can be quantified as follows. If 𝕏 is the set of all messages }} that  could be, and  is the probability of some <math>x \in \mathbb X</math>, then the entropy, , of  is defined:  

If one transmits 1000 bits (0s and 1s), and the value of each of these bits is known to the receiver (has a specific value with certainty) ahead of transmission, it is clear that no information is transmitted.  If, however, each bit is independently equally likely to be 0 or 1, 1000 shannons of information (more often called bits) have been transmitted.  Between these two extremes, information can be quantified as follows. If 𝕏 is the set of all messages }} that  could be, and  is the probability of some <math>x \in \mathbb X</math>, then the entropy, , of  is defined:  

如果一个人发送了1000比特（0s和1s），然而接收者在发送之前就已知这串比特序列中的每一个位的值，显然这个通信过程并没有任何信息（译注：如果你要告诉我一个我已已经直到的消息，那么本次通信没有传递任何信息）。但是，如果消息未知，且每个比特独立且等可能的为0或1时，则本次通信传输了1000香农的信息（通常称为“比特”）。在这两个极端之间，信息可以按以下方式进行量化。如果𝕏是{{math|''X''}}可能在的所有消息的集合{{math|{{mset|''x''₁, ..., ''x''_''n''}}}}，且{{math|''p''(''x'')}}是<math>x \in \mathbb X</math>的概率，那么熵、{{math|''H''}}和{{math|''H''}}的定义如下: <ref name = Reza>{{cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22}}</ref>

如果一个人发送了1000比特（0s和1s），然而接收者在发送之前就已知这串比特序列中的每一个位的值，显然这个通信过程并没有任何信息（译注：如果你要告诉我一个我已已经直到的消息，那么本次通信没有传递任何信息）。但是，如果消息未知，且每个比特独立且等可能的为0或1时，则本次通信传输了1000香农的信息（通常称为“比特”）。在这两个极端之间，信息可以按以下方式进行量化。如果𝕏是{{math|''X''}}可能在的所有消息的集合{{math|{''x''₁, ..., ''x''_''n''}}}，且{{math|''p''(''x'')}}是<math>x \in \mathbb X</math>的概率，那么熵、{{math|''H''}}和{{math|''H''}}的定义如下: <ref name = Reza>{{cite book | title = An Introduction to Information Theory | author = Fazlollah M. Reza | publisher = Dover Publications, Inc., New York | origyear = 1961| year = 1994 | isbn = 0-486-68210-2 | url = https://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22}}</ref>

:<math>H_{\mathrm{b}}(p) = - p \log_2 p - (1-p)\log_2 (1-p).</math>

:<math>H_{\mathrm{b}}(p) = - p \log_2 p - (1-p)\log_2 (1-p).</math>

===Joint entropy===

===Joint entropy===