Binary entropy function

In <a href="/facts/Information_theory/qNLH3U5P">information theory</a>, the binary entropy function, denoted 
 
 
 
 H
 ⁡
 (
 p
 )
 
 
 {\displaystyle \operatorname {H} (p)}
 
 or 
 
 
 
 
 H
 
 b
 
 
 ⁡
 (
 p
 )
 
 
 {\displaystyle \operatorname {H} _{\text{b}}(p)}
 
, is defined as the <a href="/facts/Entropy_(information_theory)/NLg4NLvt">entropy</a> of a <a href="/facts/Bernoulli_process/AF9E3jpu">Bernoulli process</a> (<a href="/facts/I.i.d./othIRaWt">i.i.d.</a> <a href="/facts/Binary_variable/PSFa5S2Q">binary variable</a>) with <a href="/facts/Probability/fgYvMhwb">probability</a> 
 
 
 
 p
 
 
 {\displaystyle p}
 
 of one of two values, and is given by the formula:

H
        ⁡
        (
        X
        )
        =
        −
        p
        log
        ⁡
        p
        −
        (
        1
        −
        p
        )
        log
        ⁡
        (
        1
        −
        p
        )
        .
      
    
    {\displaystyle \operatorname {H} (X)=-p\log p-(1-p)\log(1-p).}

The base of the logarithm corresponds to the choice of <a href="/facts/Units_of_information/lxjSrVJM">units of information</a>; base e corresponds to <a href="/facts/Nat_(unit)/UgNgfVgI">nats</a> and is mathematically convenient, while base 2 (<a href="/facts/Binary_logarithm/Uy7kAtRm">binary logarithm</a>) corresponds to <a href="/facts/Shannon_(unit)/VEevYzKM">shannons</a> and is conventional (as shown in the graph); explicitly:

H
        ⁡
        (
        X
        )
        =
        −
        p
        
          log
          
            2
          
        
        ⁡
        p
        −
        (
        1
        −
        p
        )
        
          log
          
            2
          
        
        ⁡
        (
        1
        −
        p
        )
        .
      
    
    {\displaystyle \operatorname {H} (X)=-p\log _{2}p-(1-p)\log _{2}(1-p).}

Note that the values at 0 and 1 are given by the limit 
 
 
 
 
 0
 log
 ⁡
 0
 :=
 
 lim
 
 x
 →
 
 0
 
 +
 
 
 
 
 x
 log
 ⁡
 x
 =
 0
 
 
 
 {\displaystyle \textstyle 0\log 0:=\lim _{x\to 0^{+}}x\log x=0}
 
 (by <a href="/facts/L%27H%C3%B4pital%27s_rule/pCXK75Io">L'Hôpital's rule</a>); and that "binary" refers to two possible values for the variable, not the units of information.
When 
 
 
 
 p
 =
 1
 
 /
 
 2
 
 
 {\displaystyle p=1/2}
 
, the binary entropy function attains its maximum value, 1 shannon (1 binary unit of information); this is the case of an <a href="/facts/Fair_coin/vTBk29k7">unbiased coin flip</a>. When 
 
 
 
 p
 =
 0
 
 
 {\displaystyle p=0}
 
 or 
 
 
 
 p
 =
 1
 
 
 {\displaystyle p=1}
 
, the binary entropy is 0 (in any units), corresponding to no information, since there is no uncertainty in the variable.

Binary entropy function open-in-new

Binary entropy function