Exact test

<h2 id="formulation">Formulation</h2>
The basic equation underlying exact tests is

Pr
        (
        
          exact
        
        )
        =
        
          ∑
          
            
              y
            
            
            :
            
            T
            (
            
              y
            
            )
            ≥
            T
            (
            
              x
              )
            
          
        
        Pr
        (
        
          y
        
        )
      
    
    {\displaystyle \Pr({\text{exact}})=\sum _{\mathbf {y} \,:\,T(\mathbf {y} )\geq T(\mathbf {x)} }\Pr(\mathbf {y} )}

where:

<ul><li>x is the actual observed outcome,</li>
<li>Pr(y) is the probability under the null hypothesis of a potentially observed outcome y,</li>
<li>T(y) is the value of the test statistic for an outcome y, with larger values of T representing cases which notionally represent greater departures from the null hypothesis,</li></ul>
and where the sum ranges over all outcomes y (including the observed one) that have the same value of the test statistic obtained for the observed sample x, or a larger one.

<h2 id="example-pearsons-chi-squared-test-versus-an-exact-test">Example: Pearson's chi-squared test versus an exact test</h2>
Main article: <a href="/facts/Pearson%27s_chi-squared_test/rGtoKuU6">Pearson's chi-squared test</a>
A simple example of this concept involves the observation that <a href="/facts/Pearson%27s_chi-squared_test/rGtoKuU6">Pearson's chi-squared test</a> is an approximate test. Suppose Pearson's chi-squared test is used to ascertain whether a six-sided die is "fair", indicating that it renders each of the six possible outcomes equally often. If the die is thrown n times, then one <a href="/facts/Expected_value/1XV0JKL8">"expects"</a> to see each outcome n/6 times. The test statistic is

∑
        
          
            
              (
              
                observed
              
              −
              
                expected
              
              
                )
                
                  2
                
              
            
            expected
          
        
        =
        
          ∑
          
            k
            =
            1
          
          
            6
          
        
        
          
            
              (
              
                X
                
                  k
                
              
              −
              n
              
                /
              
              6
              
                )
                
                  2
                
              
            
            
              n
              
                /
              
              6
            
          
        
        ,
      
    
    {\displaystyle \sum {\frac {({\text{observed}}-{\text{expected}})^{2}}{\text{expected}}}=\sum _{k=1}^{6}{\frac {(X_{k}-n/6)^{2}}{n/6}},}

where Xk is the number of times outcome k is observed. If the null hypothesis of "fairness" is true, then the <a href="/facts/Probability_distribution/EpsKKVRu">probability distribution</a> of the test statistic can be made as close as desired to the chi-squared distribution with 5 degrees of freedom by making the sample size n sufficiently large. On the other hand, if n is small, then the probabilities based on chi-squared distributions may not be sufficiently close approximations. Finding the exact probability that this test statistic exceeds a certain value would then require a <a href="/facts/Combinatorics/k14xUoKT">combinatorial enumeration</a> of all outcomes of the experiment that gives rise to such a large value of the test statistic. It is then questionable whether the same test statistic ought to be used. A <a href="/facts/Likelihood-ratio_test/oFqLh3R0">likelihood-ratio test</a> might be preferred, and the test statistic might not be a monotone function of the one above.

<h2 id="example-fishers-exact-test">Example: Fisher's exact test</h2>
Main article: <a href="/facts/Fisher%27s_exact_test/5HWUAU8G">Fisher's exact test</a>
<a href="/facts/Fisher%27s_exact_test/5HWUAU8G">Fisher's exact test</a>, based on the work of <a href="/facts/Ronald_Fisher/I2FVIpy9">Ronald Fisher</a> and <a href="/facts/E._J._G._Pitman/gxomVNka">E. J. G. Pitman</a> in the 1930s, is exact because the sampling distribution (conditional on the marginals) is known exactly. This should be compared with <a href="/facts/Pearson%27s_chi-squared_test/rGtoKuU6">Pearson's chi-squared test</a>, which (although it tests the same null) is not exact because the distribution of the test statistic is only asymptotically correct.

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Exact_statistics/DY91HFyL">Exact statistics</a></li>
<li><a href="/facts/Optimal_discriminant_analysis/BErJYblq">Optimal discriminant analysis</a></li></ul>

<ul><li><a href="/facts/Ronald_Fisher/I2FVIpy9">Ronald Fisher</a> (1954) <a href="/facts/Statistical_Methods_for_Research_Workers/aRppN873">Statistical Methods for Research Workers</a>. Oliver and Boyd.</li>
<li>Mehta, C.R.; Patel, N.R. (1998). "Exact Inference for Categorical Data". In P. Armitage and T. Colton, eds., Encyclopedia of Biostatistics, Chichester: John Wiley, pp. 1411–1422. <a href="https://resources.cytel.com/sites/default/files/resources/exact-inference-for-categorical-data.pdf">unpublished preprint</a></li>
<li>Corcoran, C. D.; Senchaudhuri, P.; Mehta, C. R.; Patel, N. R. (2005). "Exact Inference for Categorical Data". Encyclopedia of Biostatistics. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1002%2F0470011815.b2a10019">10.1002/0470011815.b2a10019</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 047084907X.</li></ul>

Exact test open-in-new

Exact test