7

From this slide, it's said that the smallest possible number of bits per symbol is as the Shannon Entropy formula defined: enter image description here

I've read this post, and still not quite understand how is this formula derived from the perspective of encoding with bits.

I'd like to get some tips like in this post, and please don't tell me that it's just because this is the only formula which satisfies the properties of a entropy function.

Thx in advance~

Community
  • 1
zhangxaochen
  • 345
  • See http://math.stackexchange.com/questions/663351/what-does-the-logpx-mean-in-the-calculation-of-entropy. – Martín-Blas Pérez Pinilla Jun 15 '14 at 12:18
  • @Martín-BlasPérezPinilla, I could agree to this formula qualitatively, but quantitatively, how to prove it's the smallest number of bits? – zhangxaochen Jun 16 '14 at 15:05
  • 2
    @zhangxaochen : That is precisely the first theorem of Shannon, it's a fundamental (simple though not trivial) result that is explained in every textbook of Information Theory, see eg. Cover- Thomas. This is too broad for getting an answer here IMO – leonbloy Jun 20 '14 at 19:35

3 Answers3

3

Suggest you read the proof that H is the only measure (up to a constant) that satisfies the axioms of information measure. It can be found here: "The Mathematical Theory of Communication" - Shannon & Weaver.

The proof of the theorem is easy to understand - only a couple of pages.

samthebest
  • 163
2

Log2 1/p is the number of bits needed to transmit symbols that occur with probability p. For example, if it occurs 1 times in 8, we need 3 bits to encode all 8 possibilities. Now just take the average number of bits weighted by p for each symbol.

Christopher Crawford
  • 21
2

Here is a simple intuitive construction of Shannon's Entropy formula: Understanding Shannon's Entropy...

Sriram V
  • 109