Design effect

<h2 id="introduction">Introduction</h2>
In <a href="/facts/Survey_methodology/vgqyWzAY">survey methodology</a>, the design effect (generally denoted as 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
, 
 
 
 
 
 D
 
 eff
 
 
 
 
 {\displaystyle D_{\text{eff}}}
 
, or 
 
 
 
 
 D
 
 eft
 
 
 2
 
 
 
 
 {\displaystyle D_{\text{eft}}^{2}}
 
) is a measure of the expected impact of a sampling design on the <a href="/facts/Variance/ULBJKXD1">variance</a> of an <a href="/facts/Estimator/CbkjcKvN">estimator</a> for some <a href="/facts/Statistical_parameter/fkJju25M">parameter</a> of a population. It is calculated as the ratio of the variance of an estimator based on a sample from an (often) complex <a href="/facts/Sampling_design/pg2wgwsg">sampling design</a>, to the variance of an alternative estimator based on a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a> (SRS) of the same number of elements.<a class="footnote-ref" id="fnref:1" href="#fn:1">1</a>: 258  The 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 (be it estimated, or known <a href="/facts/A_priori/C567SkcV">a priori</a>) can be used to evaluate the variance of an estimator in cases where the sample is not drawn using simple random sampling. It may also be useful in <a href="/facts/Sample_size_determination/sf8dAFzq">sample size calculations</a><a class="footnote-ref" id="fnref:2" href="#fn:2">2</a> and for quantifying the representativeness of samples collected with various sampling designs.
The design effect is a positive <a href="/facts/Real_number/R02gw5Pb">real number</a> that indicates an inflation (
 
 
 
 
 Deff
 
 >
 1
 
 
 {\displaystyle {\text{Deff}}>1}
 
), or deflation (
 
 
 
 
 Deff
 
 <
 1
 
 
 {\displaystyle {\text{Deff}}<1}
 
) in the <a href="/facts/Variance/ULBJKXD1">variance</a> of an estimator for some parameter, that is due to the study not using SRS (with 
 
 
 
 
 Deff
 
 =
 1
 
 
 {\displaystyle {\text{Deff}}=1}
 
, when the variances are identical).<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a>: 53, 54  Intuitively we can get 
 
 
 
 
 Deff
 
 <
 1
 
 
 {\displaystyle {\text{Deff}}<1}
 
 when we have some a-priori knowledge we can exploit during the sampling process (which is somewhat rare). And, in contrast, we often get 
 
 
 
 
 Deff
 
 >
 1
 
 
 {\displaystyle {\text{Deff}}>1}
 
 when we need to compensate for some limitation in our ability to collect data (which is more common). Some sampling designs that could introduce 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 generally greater than 1 include: <a href="/facts/Cluster_sampling/2PX8m0fd">cluster sampling</a> (such as when there is <a href="/facts/Correlation/egkluAEm">correlation</a> between observations), <a href="/facts/Stratified_sampling/CvAUCegT">stratified sampling</a> (with disproportionate allocation to the strata sizes), <a href="/facts/Cluster_randomized_controlled_trial/NsQxXCwH">cluster randomized controlled trial</a>, disproportional (unequal probability) sample (e.g. <a href="/facts/Poisson_sampling/3ha7k2y7">Poisson sampling</a>), statistical adjustments of the data for non-coverage or non-response, and many others. <a href="/facts/Stratified_sampling/CvAUCegT">Stratified sampling</a> can yield 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 that is smaller than 1 when using Proportionate allocation to strata sizes (when these are known a-priori, and correlated to the outcome of interest) or Optimum allocation (when the variance differs between strata and is known a-priori).
Many calculations (and estimators) have been proposed in the literature for how a known sampling design influences the variance of estimators of interest, either increasing or decreasing it. Generally, the design effect varies among different statistics of interests, such as the total or <a href="/facts/Ratio_distribution/5rUEmWkG">ratio mean</a>. It also matters if the sampling design is correlated with the outcome of interest. For example, a possible sampling design might be such that each element in the sample may have a different probability to be selected. In such cases, the level of correlation between the probability of selection for an element and its measured outcome can have a direct influence on the subsequent design effect. Lastly, the design effect can be influenced by the distribution of the outcome itself. All of these factors should be considered when estimating and using design effect in practice.<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a>: 13 

<h2 id="history">History</h2>
The term "design effect" was coined by <a href="/facts/Leslie_Kish/oy1EYwHC">Leslie Kish</a> in his 1965 book "Survey Sampling."<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a>: 88, 258  In it, Kish proposed the general definition for the design effect,<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a> as well as formulas for the design effect of cluster sampling (with intraclass correlation);<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a>: 162  and the famous design effect formula for unequal probability sampling.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a>: 427  These are often known as "Kish's design effect", and were later combined into a single formula.
In a 1995 paper,<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>: 73  Kish mentions that a similar concept, termed "Lexis ratio", was described at the end of the 19th century. The closely related <a href="/facts/Intraclass_correlation/GjhrGfUD">Intraclass correlation</a> was described by <a href="/facts/Ronald_Fisher/I2FVIpy9">Fisher</a> in 1950, while computations of ratios of variances were already published by Kish and others from the late 1940s to the 1950s. One of the precursors to Kish's definition was work done by Cornfield in 1951.<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a><a class="footnote-ref" id="fnref:11" href="#fn:11">11</a>
In his 1995 paper, Kish proposed that considering the design effect is necessary when averaging the same measured quantity from multiple surveys conducted over a period of time.<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a>: 57–62  He also suggested that the design effect should be considered when extrapolating from the error of simple statistics (e.g. the mean) to more complex ones (e.g. regression coefficients). However, when analyzing data (e.g., using survey data to fit models), 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 values are less useful nowadays due to the availability of specialized software for analyzing survey data. Prior to the development of software that computes standard errors for many types of designs and estimates, analysts would adjust standard errors produced by software that assumed all records in a dataset were i.i.d by multiplying them by a 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 (see Deft definition below).

<h2 id="definitions">Definitions</h2>
<h3>Notations</h3>
Table 1: Summary of notation<table><tbody><tr><th>Symbol</th><th>Description</th></tr><tr><td> var ( θ ^ w ) {\displaystyle {\text{var}}({\hat {\theta }}_{w})} </td><td>Variance of an estimator θ ^ w {\displaystyle {\hat {\theta }}_{w}} under a given sampling design</td></tr><tr><td> var ( θ ^ S R S W O R ) {\displaystyle {\text{var}}({\hat {\theta }}_{SRSWOR})} </td><td>Variance of an estimator θ ^ S R S W O R {\displaystyle {\hat {\theta }}_{SRSWOR}} under simple random sampling without replacement (SRSWOR)</td></tr><tr><td> var ( θ ^ S R S W R ) {\displaystyle {\text{var}}({\hat {\theta }}_{SRSWR})} </td><td>Variance of an estimator θ ^ S R S W R {\displaystyle {\hat {\theta }}_{SRSWR}} under simple random sampling with replacement (SRSWR)</td></tr><tr><td> Deff {\displaystyle {\text{Deff}}} , D eff {\displaystyle D_{\text{eff}}} </td><td>Design effect, a measure of the impact of a sampling design on the variance of an estimator compared to simple random sampling without replacement (SRSWOR), Deff p ( θ ^ ) = v a r ( θ ^ w ) v a r ( θ ^ S R S W O R ) {\displaystyle {\text{Deff}}_{p}({\hat {\theta }})={\frac {var({\hat {\theta }}_{w})}{var({\hat {\theta }}_{SRSWOR})}}} </td></tr><tr><td> Deft {\displaystyle {\text{Deft}}} , D eft {\displaystyle D_{\text{eft}}} </td><td>Design effect factor, the square root of the ratio of variances under a given sampling design and SRS with replacement (SRSWR), Deft = var ( θ ^ w ) var ( θ ^ S R S W R ) {\displaystyle {\text{Deft}}={\sqrt {\frac {{\text{var}}({\hat {\theta }}_{w})}{{\text{var}}({\hat {\theta }}_{SRSWR})}}}} </td></tr><tr><td> n {\displaystyle n} </td><td>Sample size</td></tr><tr><td> N {\displaystyle N} </td><td>Population size</td></tr><tr><td> n eff {\displaystyle n_{\text{eff}}} </td><td>Effective sample size, the sample size under SRS needed to achieve the same variance as the given sampling design, n eff = n Deff {\displaystyle n_{\text{eff}}={\frac {n}{\text{Deff}}}} </td></tr><tr><td> w i {\displaystyle w_{i}} </td><td>Weight for the i {\displaystyle i} -th unit</td></tr><tr><td> n h {\displaystyle n_{h}} </td><td>Sample size for stratum h {\displaystyle h} </td></tr><tr><td> N h {\displaystyle N_{h}} </td><td>Population size for stratum h {\displaystyle h} </td></tr><tr><td> w h {\displaystyle w_{h}} </td><td>Weight for stratum h {\displaystyle h} </td></tr><tr><td> H {\displaystyle H} </td><td>Total number of strata</td></tr><tr><td> n ∗ {\displaystyle n^{*}} , b ¯ {\displaystyle {\bar {b}}} </td><td>Average cluster size</td></tr><tr><td> K {\displaystyle K} </td><td>Total number of clusters</td></tr><tr><td> n k {\displaystyle n_{k}} </td><td>Sample size for cluster k {\displaystyle k} </td></tr><tr><td> ρ {\displaystyle \rho } </td><td>Intraclass correlation coefficient (ICC) for cluster sampling</td></tr><tr><td> L {\displaystyle L} , C V 2 {\displaystyle C_{V}^{2}} , relvar ( w ) {\displaystyle {\text{relvar}}(w)} </td><td>Measures of variation in weights using the coefficient of variation (CV) squared (relvariance)</td></tr><tr><td> ρ ^ y , P {\displaystyle {\hat {\rho }}_{y,P}} </td><td>Estimated correlation between the outcome variable y {\displaystyle y} and the selection probabilities P {\displaystyle P} </td></tr><tr><td> α ^ {\displaystyle {\hat {\alpha }}} </td><td>Estimated intercept in the linear regression of the outcome variable y {\displaystyle y} on the selection probabilities P {\displaystyle P} </td></tr><tr><td> σ ^ y {\displaystyle {\hat {\sigma }}_{y}} </td><td>Estimated standard deviation of the outcome variable y {\displaystyle y} </td></tr><tr><td> c v w {\displaystyle cv_{w}} , c v P {\displaystyle cv_{P}} </td><td>Coefficient of variation for the weights w {\displaystyle w} and selection probabilities P {\displaystyle P} , respectively</td></tr><tr><td> f {\displaystyle f} </td><td>Sampling fraction, f = n / N {\displaystyle f=n/N} </td></tr><tr><td> S y 2 {\displaystyle S_{y}^{2}} </td><td>Population variance of the outcome variable y {\displaystyle y} </td></tr><tr><td> p i {\displaystyle p_{i}} , P i {\displaystyle P_{i}} </td><td>Selection probability for the i {\displaystyle i} -th unit</td></tr><tr><td> π i {\displaystyle \pi _{i}} </td><td>Inclusion probability for the i {\displaystyle i} -th unit</td></tr></tbody></table>
<h3>Deff</h3>
The design effect, commonly denoted by 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 (or 
 
 
 
 
 D
 
 eff
 
 
 
 
 {\displaystyle D_{\text{eff}}}
 
, sometimes with additional subscripts), is the ratio of two theoretical variances for <a href="/facts/Estimator/CbkjcKvN">estimators</a> of some <a href="/facts/Statistical_parameter/fkJju25M">parameter</a> (
 
 
 
 θ
 
 
 {\displaystyle \theta }
 
):<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a><a class="footnote-ref" id="fnref:14" href="#fn:14">14</a>

<ul><li>The numerator represents the actual variance for an estimator of a parameter (
 
 
 
 
 
 
 
 θ
 ^
 
 
 
 
 w
 
 
 
 
 {\displaystyle {\hat {\theta }}_{w}}
 
) under a given sampling design 
 
 
 
 p
 
 
 {\displaystyle p}
 
;</li>
<li>The denominator represents the variance assuming the same sample size, but if the sample were obtained using the estimator for <a href="/facts/Simple_random_sampling/vDnCHnNx">simple random sampling</a> without replacement (
 
 
 
 
 
 
 
 θ
 ^
 
 
 
 
 S
 R
 S
 W
 O
 R
 
 
 
 
 {\displaystyle {\hat {\theta }}_{SRSWOR}}
 
).</li></ul>
So that:

Deff
          
          
            p
          
        
        (
        
          
            
              θ
              ^
            
          
        
        )
        =
        
          
            
              v
              a
              r
              (
              
                
                  
                    
                      θ
                      ^
                    
                  
                
                
                  w
                
              
              )
            
            
              v
              a
              r
              (
              
                
                  
                    
                      θ
                      ^
                    
                  
                
                
                  S
                  R
                  S
                  W
                  O
                  R
                
              
              )
            
          
        
      
    
    {\displaystyle {\text{Deff}}_{p}({\hat {\theta }})={\frac {var({\hat {\theta }}_{w})}{var({\hat {\theta }}_{SRSWOR})}}}

In other words, 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 measures the extent to which the variance has increased (or, in some cases, decreased) because the sample was drawn and adjusted to a specific sampling design (e.g., using weights or other measures) compared to if the sample was from a <a href="/facts/Simple_random_sampling/vDnCHnNx">simple random sample</a> (without replacement). Notice how the definition of 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 is based on parameters of the population that are often unknown, and that are hard to estimate directly. Specifically, the definition involves the variances of estimators under two different sampling designs, even though only a single sampling design is used in practice.
For example, when estimating the population mean, the 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 (for some sampling design p) is:<a class="footnote-ref" id="fnref:15" href="#fn:15">15</a>: 4 <a class="footnote-ref" id="fnref:16" href="#fn:16">16</a>: 54 <a class="footnote-ref" id="fnref:17" href="#fn:17">17</a>

Deff
          
          
            p
          
        
        =
        
          
            
              v
              a
              
                r
                
                  p
                
              
              (
              
                
                  
                    
                      y
                      ¯
                    
                  
                
                
                  p
                
              
              )
            
            
              (
              1
              −
              f
              )
              
                S
                
                  y
                
                
                  2
                
              
              
                /
              
              n
            
          
        
      
    
    {\displaystyle {\text{Deff}}_{p}={\frac {var_{p}({\bar {y}}_{p})}{(1-f)S_{y}^{2}/n}}}

Where 
 
 
 
 n
 
 
 {\displaystyle n}
 
 is the sample size, 
 
 
 
 f
 =
 n
 
 /
 
 N
 
 
 {\displaystyle f=n/N}
 
 is the fraction of the sample from the population, 
 
 
 
 (
 1
 −
 f
 )
 
 
 {\displaystyle (1-f)}
 
 is the (squared) <a href="/facts/Standard_error/Iv96AyCD">finite population correction</a> (FPC), 
 
 
 
 
 S
 
 y
 
 
 2
 
 
 
 
 {\displaystyle S_{y}^{2}}
 
 is the <a href="/facts/Variance/ULBJKXD1">unbiassed sample variance</a>, and 
 
 
 
 v
 a
 
 r
 
 p
 
 
 (
 
 
 
 
 y
 ¯
 
 
 
 
 p
 
 
 )
 
 
 {\displaystyle var_{p}({\bar {y}}_{p})}
 
 is some estimator of the variance of the mean under the sampling design. The issue with the above formula is that it is extremely rare to be able to directly estimate the variance of the estimated mean under two different sampling designs, since most studies rely on only a single sampling design.
There are many ways of calculation 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
, depending on the parameter of interest (e.g. population total, population mean, quantiles, ratio of quantities etc.), the estimator used, and the sampling design (e.g. clustered sampling, stratified sampling, post-stratification, multi-stage sampling, etc.).<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>: 98  The process of estimating 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 for specific designs will be described in the following section.

<h3>Deft</h3>
A related quantity to 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
, proposed by Kish in 1995, is the Design Effect Factor, abbreviated as 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 (or also 
 
 
 
 
 D
 
 eft
 
 
 
 
 {\displaystyle D_{\text{eft}}}
 
).<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a>: 56 <a class="footnote-ref" id="fnref:20" href="#fn:20">20</a> It is defined as the square root of the variance ratios while also having the denominator use a simple random sample with replacement (SRSWR), instead of without replacement (SRSWOR):

 
 
 
 
 Deft
 
 =
 
 
 
 
 
 var
 
 (
 
 
 
 
 θ
 ^
 
 
 
 
 w
 
 
 )
 
 
 
 var
 
 (
 
 
 
 
 θ
 ^
 
 
 
 
 S
 R
 S
 W
 R
 
 
 )
 
 
 
 
 
 
 {\displaystyle {\text{Deft}}={\sqrt {\frac {{\text{var}}({\hat {\theta }}_{w})}{{\text{var}}({\hat {\theta }}_{SRSWR})}}}}

In this later definition (proposed in 1995, vs 1965) Kish argued in favor of using 
 
 
 
 
 
 Deft
 
 
 2
 
 
 
 
 {\displaystyle {\text{Deft}}^{2}}
 
 over 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 for several reasons. It was argued that SRS "without replacement" (with its positive effect on the variance) should be captured in the denominator part in the definition of the design effect, since it is part of the sampling design. Also, since often the use of the factor is in <a href="/facts/Confidence_interval/NS8mG5UE">confidence intervals</a>), it was claimed that using 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 will be simpler than writing 
 
 
 
 
 
 Deff
 
 
 
 
 {\displaystyle {\sqrt {\text{Deff}}}}
 
. It is also said that for many cases when the population is very large, 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 is (almost) the square root of 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 (
 
 
 
 
 Deft
 
 ≈
 
 
 Deff
 
 
 
 
 {\displaystyle {\text{Deft}}\approx {\sqrt {\text{Deff}}}}
 
), hence it is easier to use than exactly calculating the <a href="/facts/Standard_error/Iv96AyCD">finite population correction</a> (FPC).<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a>
Even so, in various cases a researcher might approximate the 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 by calculating the variance in the numerator while assuming SRS with replacement (SRSWR) instead of SRS without replacement (SRSWOR), even if it is not precise. For example, consider a multistage design with primary sampling units (PSUs) selected systematically with probability proportional to some measure of size from a list sorted in a particular way (say, by number of households in each PSU). Also, let it be combined with an estimator that uses <a href="/facts/Raking/dM0J5zQc">raking</a> to match the totals for several demographic variables. In such a design, the joint selection probabilities for the PSUs, which are needed for a without replacement variance estimator, are 0 for some pairs of PSUs - implying that an exact design-based (i.e., repeated sampling) variance estimator does not exist. Another example is when a public use file issued by some government agency is used for analysis. In such a case the information on joint selection probabilities of first-stage units is almost never released. As a result, an analyst cannot estimate a with replacement variance for the numerator even if desired. The standard workaround is to compute a variance estimator as if the PSUs were selected with replacement. This is the default choice in software packages such as Stata, the R survey package, and the SAS survey procedures.

<h3>Effective sample size</h3>
The effective sample size, defined by Kish in 1965, is calculated by dividing the original sample size by the design effect.<a class="footnote-ref" id="fnref:22" href="#fn:22">22</a>: 162, 259 <a class="footnote-ref" id="fnref:23" href="#fn:23">23</a>: 190, 192  Namely:

n
          
            eff
          
        
        =
        
          
            n
            Deff
          
        
      
    
    {\displaystyle n_{\text{eff}}={\frac {n}{\text{Deff}}}}

This quantity reflects what would be the sample size that is needed to achieve the current variance of the estimator (for some parameter) with the existing design, if the sample design (and its relevant parameter estimator) were based on a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>.<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a>
A related quantity is the effective sample size ratio, which can be calculated by simply taking the inverse of 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 (i.e., 
 
 
 
 
 
 
 n
 
 eff
 
 
 n
 
 
 =
 
 
 1
 Deff
 
 
 
 
 {\displaystyle {\frac {n_{\text{eff}}}{n}}={\frac {1}{\text{Deff}}}}
 
).
For example, let the design effect, for estimating the population mean based on some sampling design, be 2. If the sample size is 1,000, then the effective sample size will be 500. It means that the variance of the <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">weighted mean</a> based on 1,000 samples will be the same as that of a simple <a href="/facts/Arithmetic_mean/L3Pv333o">mean</a> based on 500 samples obtained using a simple random sample.

<h2 id="the-design-effect-for-well-known-sampling-designs">The design effect for well-known sampling designs</h2>
<h3>The design effect depends on sampling design and statistical adjustments</h3>
Different sampling designs and statistical adjustments may have substantially different impact on the bias and variance of estimators (such as the mean).
An example of a design which can lead to estimation efficiency, compared to simple random sampling, is <a href="/facts/Stratified_sampling/CvAUCegT">Stratified sampling</a>. This efficiency is gained by leveraging information about the composition of the population. For example, if it is known that gender is correlated with the outcome of interest, and also that the male-female ratio for some population is (say) 50%-50%, then sampling exactly half of the sample from each gender will reduce the variance of the outcome's estimator. Similarly, if a particular sub-population is of special interest, deliberately over-sampling from that sub-population will decrease the variance for estimations made about it.
Improvement in variance efficiency might sometimes be sacrificed for convenience or cost. For example, in the <a href="/facts/Cluster_sampling/2PX8m0fd">cluster sampling</a> case the units may have equal or unequal selection probabilities, irrespective of their <a href="/facts/Intra-class_correlation/GjhrGfUD">intra-class correlation</a> (and their negative effect of increasing the variance of the estimators). We might decide (for practical reasons) to collect responses from only 2 people of each household (i.e., a sampled cluster), which could lead to more complex post-sampling adjustment to deal with unequal selection probabilities. Also, such decisions could lead to less efficient estimators than just taking a fixed proportion of responses from a cluster.
When the sampling design isn’t set in advance and needs to be figured out from the data we have, this can lead to an increase of both the variance and bias of the weighted estimator. This might happen when making adjustments for issues like non-coverage, non-response, or an unexpected strata split of the population that wasn’t available during the initial sampling stage. In these cases, we might use statistical procedures such as post-stratification, raking, or inverse propensity score weighting (where the propensity scores are estimated), among other methods. Using these methods requires assumptions about the initial design model. For example, when we use post-stratification based on age and gender, it is assumed that these variables can explain a significant portion of the bias in the sample. The quality of these estimators is closely tied to the quality of the additional information and the <a href="/facts/Missing_data/p2pKMXRV">missing at random</a> assumptions used when making them. Either way, even when estimators (like propensity score models) do a good job capturing most of the sampling design, using the weights can make a small or a large difference, depending on the specific data-set.
Due to the large variety in sampling designs (with or without an effect on unequal selection probabilities), different formulas have been developed to capture the potential design effect, as well as to estimate the variance of estimators when accounting for the sampling designs.<a class="footnote-ref" id="fnref:25" href="#fn:25">25</a> Sometimes, these different design effects can be compounded together (as in the case of unequal selection probability and cluster sampling, more details in the following sections). Whether or not to use these formulas, or just assume SRS, depends on the expected amount of bias reduction vs. the increase in estimator variance (and in the overhead of methodological and technical complexity).<a class="footnote-ref" id="fnref:26" href="#fn:26">26</a>: 426 

Table 2: Summary of design effect formulas<table><tbody><tr><th>Formula Name</th><th>Equation</th><th>Description</th></tr><tr><td>Kish's design effect for unequal weights</td><td> Deff = n ∑ i = 1 n w i 2 ( ∑ i = 1 n w i ) 2 = w 2 ¯ w ¯ 2 {\displaystyle {\text{Deff}}={\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}={\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}} </td><td>Measures the loss in precision due to unequal weights, where w i {\displaystyle w_{i}} is the weight for the i {\displaystyle i} -th unit.</td></tr><tr><td>Kish's design effect for cluster sampling</td><td> Deff = 1 + ( n ∗ − 1 ) ρ {\displaystyle {\text{Deff}}=1+(n^{*}-1)\rho } </td><td>Measures the loss in precision due to cluster sampling, where n ∗ {\displaystyle n^{*}} is the average cluster size and ρ {\displaystyle \rho } is the intraclass correlation.</td></tr><tr><td>Kish's combined design effect</td><td> Deff = n ∑ h = 1 H ( n h w h 2 ) ( ∑ h = 1 H n h w h ) 2 ( 1 + ( n ∗ − 1 ) ρ ) {\displaystyle {\text{Deff}}={\frac {n\sum _{h=1}^{H}(n_{h}w_{h}^{2})}{(\sum _{h=1}^{H}n_{h}w_{h})^{2}}}(1+(n^{*}-1)\rho )} </td><td>Measures the combined effect of unequal weights and cluster sampling, where n h {\displaystyle n_{h}} and w h {\displaystyle w_{h}} are the sample size and weight for the h {\displaystyle h} -th stratum, respectively.</td></tr><tr><td>Spencer's design effect for estimated total</td><td> Deff = ( 1 − ρ ^ y , P 2 ) ( 1 + L ) + ( α ^ σ ^ y ) 2 L {\displaystyle {\text{Deff}}=(1-{\hat {\rho }}_{y,P}^{2})(1+L)+({\frac {\hat {\alpha }}{{\hat {\sigma }}_{y}}})^{2}L} </td><td>Measures the design effect for estimating a total when there is a correlation between the outcome and the selection probabilities, where ρ ^ y , P {\displaystyle {\hat {\rho }}_{y,P}} is the estimated correlation, L {\displaystyle L} is the relvariance of the weights, α ^ {\displaystyle {\hat {\alpha }}} is the estimated intercept, and σ ^ y {\displaystyle {\hat {\sigma }}_{y}} is the estimated standard deviation of the outcome.</td></tr><tr><td>Park and Lee's design effect for estimated ratio mean</td><td> Deff = ( 1 − ρ ^ y , P 2 ) ( 1 + c v w 2 ) + ρ ^ y , P 2 c v P 2 c v w 2 {\displaystyle {\text{Deff}}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+{\frac {{\hat {\rho }}_{y,P}^{2}}{cv_{P}^{2}}}cv_{w}^{2}} </td><td>Measures the design effect for estimating a ratio mean when there is a correlation between the outcome and the selection probabilities, where c v w {\displaystyle cv_{w}} and c v P {\displaystyle cv_{P}} are the coefficients of variation for the weights and selection probabilities, respectively.</td></tr><tr><td>Henry's design effect for calibration weighting</td><td>Extends Kish's design effect to include calibration weighting in single-stage samples</td><td>Proposes a model-assisted design effect measure for single-stage sampling with calibration weighting, considering the correlation between the outcome and the calibration variables.</td></tr><tr><td>Lohr's design effect for regression slope</td><td>Provides design effect formulas for OLS and GLS regression slope estimators in cluster sampling</td><td>Presents design effect formulas for ordinary least squares (OLS) and generalized least squares (GLS) regression slope estimators in the context of cluster sampling, using a random coefficient model.</td></tr></tbody></table>
<h3>Unequal selection probabilities</h3>
<h4>Sources of unequal selection probabilities</h4>
Table 3: Summary of sources for unequal selection probabilities<table><tbody><tr><th>Source</th><th>Description</th><th>Examples</th><th>Impact on Sampling Probabilities</th></tr><tr><td>Disproportional sampling</td><td>Deliberately over/under sampling specific sub-populations or clusters</td><td>- Optimum allocation in stratified sampling- Oversampling smaller groups for comparison- Cluster sampling with unequal cluster sizes</td><td>Leads to unequal selection probabilities by design</td></tr><tr><td>Non-coverage</td><td>Failure to include all elements of the target population in the sampling frame</td><td>- Sampling based on incomplete lists (e.g., phone books)- Advertising to recruit survey participants</td><td>Affects sampling probabilities, but the impact is difficult to measure and adjust for</td></tr><tr><td>Non-response</td><td>Failure to obtain measurements from sampled units that were intended to be measured</td><td>- Unit non-response (e.g., refusal, not-at-home)- Item non-response (e.g., sensitive questions)- Inability to respond (e.g., language barrier, illness)</td><td>Leads to unequal selection probabilities, as non-response rates may vary across subgroups</td></tr><tr><td>Statistical adjustments</td><td>Post-hoc adjustments to the sample weights to account for known population characteristics or to mitigate non-coverage and non-response biases</td><td>- Post-stratification- Raking- Propensity score weighting- Calibration weighting</td><td>Introduces unequal weights to improve representativeness, but may increase variance</td></tr></tbody></table>
There are various ways to sample units so that each unit would have the exact same probability of selection. Such methods are called <a href="/facts/Simple_random_sample/vDnCHnNx">equal probability sampling</a> (EPSEM) methods. Some of the more basic methods include <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sampling</a> (SRS, with or without replacement) and <a href="/facts/Systematic_sampling/KQhBEuXI">systematic sampling</a> for getting a fixed sample size. There is also <a href="/facts/Bernoulli_sampling/N7qeq0tb">Bernoulli sampling</a> with a random sample size. More advanced techniques such as <a href="/facts/Stratified_sampling/CvAUCegT">stratified sampling</a> and <a href="/facts/Cluster_sampling/2PX8m0fd">cluster sampling</a> can also be designed to be EPSEM. For example, in cluster sampling we can use a two stage sampling in which we sample each cluster (which may be of different sizes) with equal probability, and then sample from each cluster at the second stage using SRS with a fixed proportion (e.g. sample half of the cluster, the whole cluster, etc.). This method will yield EPSEM, but the specific number of elements we end up with is stochastic (i.e., non deterministic).<a class="footnote-ref" id="fnref:27" href="#fn:27">27</a><a class="footnote-ref" id="fnref:28" href="#fn:28">28</a>: 3–8  Another strategy for cluster sampling that leads to EPSEM is to sample clusters in a way that is proportional to their sizes, and then sample a fixed number of elements inside each cluster.<a class="footnote-ref" id="fnref:29" href="#fn:29">29</a>
In their works, <a href="/facts/Leslie_Kish/oy1EYwHC">Kish</a> and others highlight several known reasons that lead to unequal selection probabilities:<a class="footnote-ref" id="fnref:30" href="#fn:30">30</a>: 425 <a class="footnote-ref" id="fnref:31" href="#fn:31">31</a>: 185 <a class="footnote-ref" id="fnref:32" href="#fn:32">32</a>: 69 <a class="footnote-ref" id="fnref:33" href="#fn:33">33</a>: 50, 395 <a class="footnote-ref" id="fnref:34" href="#fn:34">34</a>: 306 

<ol><li>Disproportional sampling due to selection frame or procedure. This happens when a researcher deliberately over- or under-samples specific sub-populations or clusters. For example:
<ul><li>In <a href="/facts/Stratified_sampling/CvAUCegT">stratified sampling</a> when units from some strata are known to have a larger variance than other strata. In such cases, the intention of the researcher may be to use this prior knowledge about the variance between strata in order to reduce the overall variance of an estimator of some population level parameter of interest (e.g., the mean). This can be achieved by a strategy known as <a href="/facts/Sample_size_determination/sf8dAFzq">optimum allocation</a>, in which a stratum 
 
 
 
 h
 
 
 {\displaystyle h}
 
 is over sampled proportional to higher standard deviation and lower sampling cost (i.e., 
 
 
 
 
 f
 
 h
 
 
 ∝
 
 
 
 S
 
 h
 
 
 
 
 C
 
 h
 
 
 
 
 
 
 
 {\displaystyle f_{h}\propto {\frac {S_{h}}{\sqrt {C_{h}}}}}
 
, where 
 
 
 
 
 S
 
 h
 
 
 
 
 {\displaystyle S_{h}}
 
 is the standard deviation of the outcome in 
 
 
 
 h
 
 
 {\displaystyle h}
 
, and 
 
 
 
 
 C
 
 h
 
 
 
 
 {\displaystyle C_{h}}
 
 relates to the cost of recruiting one element from 
 
 
 
 h
 
 
 {\displaystyle h}
 
). An example of an optimum allocation is Neyman's optimal allocation which, when cost is fixed for recruiting people from each stratum, the sample size is: 
 
 
 
 
 n
 
 h
 
 
 =
 n
 
 
 
 
 W
 
 h
 
 
 
 S
 
 U
 h
 
 
 
 
 
 ∑
 
 h
 
 
 
 W
 
 h
 
 
 
 S
 
 U
 h
 
 
 
 
 
 
 
 {\displaystyle n_{h}=n{\frac {W_{h}S_{Uh}}{\sum _{h}W_{h}S_{Uh}}}}
 
. Where the summation is over all strata: n is the total sample size; 
 
 
 
 
 n
 
 h
 
 
 
 
 {\displaystyle n_{h}}
 
 is the sample size for stratum h; 
 
 
 
 
 W
 
 h
 
 
 =
 
 
 
 N
 
 h
 
 
 N
 
 
 
 
 {\displaystyle W_{h}={\frac {N_{h}}{N}}}
 
 is the relative size of stratum h as compared to the entire population N; and 
 
 
 
 
 S
 
 U
 h
 
 
 
 
 {\displaystyle S_{Uh}}
 
 is the standard error in stratum h.<a class="footnote-ref" id="fnref:35" href="#fn:35">35</a> A related concept to optimum design is <a href="/facts/Optimal_design/qCH9pPt5">optimal experimental design</a>.</li>
<li>If there is interest in comparing two strata (e.g., people from two specific socio-demographic groups, or from two regions, etc.), in which case the smaller group may be over-sampled. This way, the variance of the estimator that compares the two groups is reduced.</li>
<li>In cluster sampling there may be clusters of different sizes but the procedure samples from all clusters using <a href="/facts/Simple_random_sample/vDnCHnNx">SRS</a>, and all elements in the cluster are measured (for example, if the cluster sizes are not known upfront at the stage of sampling).</li>
<li>In some two-stage cluster sampling based cluster sizes. For example, when in the first stage the clusters are sampled proportionally to the estimation of their size (a.k.a.: PPS Probability Proportional to Size) and at the second stage a fixed proportion of elements are chosen (e.g., half, or all the elements in the cluster) - then the selection probabilities are different for elements from different clusters. A similar case is when the first stage attempts to sample the clusters using PPS, the second stage uses a fixed number of elements in each cluster - but the cluster sizes used for the first stage sampling were inaccurate (so that some smaller cluster may have a higher-than-it-should chance of being selected. And vice versa for larger clusters with too-small a chance of being sampled). In such cases, the larger the errors in the sampling probabilities used in the first stage, the larger the unequal selection probabilities for each element will be.<a class="footnote-ref" id="fnref:36" href="#fn:36">36</a>: 109 <a class="footnote-ref" id="fnref:37" href="#fn:37">37</a></li>
<li>When the frame used for sampling includes duplication of some of the items, thus leading some items to have a larger probability than others to be sampled (e.g., if the sampling frame was created by merging several lists. Or if recruiting users from several ad channels in which some of the users are available for recruitment from several of the channels, while others are available to be recruited from only one of the channels) so that different units would have different sampling probabilities, thus making this sampling procedure to not be EPSEM.<a class="footnote-ref" id="fnref:38" href="#fn:38">38</a>: 3–8 <a class="footnote-ref" id="fnref:39" href="#fn:39">39</a>: 186 </li>
<li>When several different samples/frames are to be combined. For example, if running different ad campaigns for recruiting respondents. Or when combining results from several studies done by different researchers and/or at different times (i.e., <a href="/facts/Meta-analysis/IcfChd8z">Meta-analysis</a>).<a class="footnote-ref" id="fnref:40" href="#fn:40">40</a>: 188 </li></ul>
When disproportional sampling happens, due to sampling design decisions, the researcher may (sometimes) be able to trace back the decision and accurately calculate the exact inclusion probability. When these selection probabilities are hard to trace back, they may be estimated using some propensity score model combined with information from auxiliary variables (e.g., age, gender, etc.).</li>
<li>Non-coverage.<a class="footnote-ref" id="fnref:41" href="#fn:41">41</a>: 527, 528  This happens, for example, if people are sampled based on some pre-defined list that doesn't include all the people in the population (e.g., a phone book or using ads to recruit people to a survey). These missing units are missing due to some failure of creating the <a href="/facts/Sampling_frame/LhrXV2FS">sampling frame</a>, as opposed to deliberate exclusion of some people (e.g. minors, people who cannot vote, etc.). The effect of non-coverage on sampling probability is considered difficult to measure (and adjust for) in various survey situations, unless strong assumptions are made. Adjustments for non-coverage can lead to inadequate weights when the relevant covariates are not used for adjustment. If there are covariates that can be used to correct for non-coverage, they are expected to lead to unequal survey weights.</li>
<li>Non-response. This refers to the failure of obtaining measurements on sampled units that are intended to be measured. Reasons for non-response are varied and depend on the context. A person may be temporarily unavailable, for example if they are not available to answer the phone when a telephone survey is done. A person may also refuse to answer the survey due to a variety of reasons, e.g. different tendencies of people from different ethnic/demographic/socio-economic groups to respond in general; insufficient incentive to spend the time or share data; the identity of the institution that is running the survey; inability to respond (e.g. due to illness, illiteracy, or a language barrier); respondent is not found (e.g. they moved); the response was lost/destroyed during encoding or transmission (i.e., measurement error). In the context of surveys, these reasons may be related to answering the entire survey or just specific questions.<a class="footnote-ref" id="fnref:42" href="#fn:42">42</a>: 532 <a class="footnote-ref" id="fnref:43" href="#fn:43">43</a>: 186 </li>
<li>Statistical adjustments. These may include methods such as <a href="/facts/Sampling_(statistics)/kIb01xdL">post-stratification</a>, <a href="/facts/Raking/dM0J5zQc">raking</a>, or <a href="/facts/Propensity_score_matching/ajVY6GqP">propensity score (estimation) models</a> - used to perform an adjustment of the sample to some known (or estimated) strata sizes. These adjustments can be in addition of design weights, which aims to account for imbalances due to some known sampling design. Such procedures are used to mitigate issues in the sampling ranging from <a href="/facts/Sampling_error/3k8Fjlb3">sampling error</a>, under-coverage of the sampling frame to non-response.<a class="footnote-ref" id="fnref:44" href="#fn:44">44</a>: 45 <a class="footnote-ref" id="fnref:45" href="#fn:45">45</a> For example, these methods can be used to make the sample more similar to some target "controls" (i.e., population of interest), a process also called "standardization".<a class="footnote-ref" id="fnref:46" href="#fn:46">46</a>: 187  In such cases, these adjustments help with providing unbiased estimators (often with the cost of increased variance, as seen in the following sections). If the original sample is a <a href="/facts/Nonprobability_sampling/y8ngS487">nonprobability sample</a>, then post-stratification adjustments are just similar to <a href="/facts/Quota_sampling/rcDcc2fr">quota sampling</a>.<a class="footnote-ref" id="fnref:47" href="#fn:47">47</a>: 188, 189  Note that if a simple random sample is used, a post-stratification (using some auxiliary information) does not offer an estimator that is uniformly better than just an unweighted estimator. However, it can be viewed as a more "robust" estimator.<a class="footnote-ref" id="fnref:48" href="#fn:48">48</a> Alternatively, when the sampling design is fully known (leading to some 
 
 
 
 
 p
 
 h
 
 
 
 
 {\displaystyle p_{h}}
 
 probability of selection for some element from stratum h), and the non-response is measurable (i.e., we know that only 
 
 
 
 
 r
 
 h
 
 
 
 
 {\displaystyle r_{h}}
 
 observations answered in stratum h), then an exactly known <a href="/facts/Inverse_probability_weighting/SR1Fy0W7">inverse probability weight</a> can be calculated for each element i from stratum h using: 
 
 
 
 
 w
 
 i
 
 
 =
 
 
 1
 
 
 p
 
 h
 
 
 
 r
 
 h
 
 
 
 
 
 
 
 {\displaystyle w_{i}={\frac {1}{p_{h}r_{h}}}}
 
.<a class="footnote-ref" id="fnref:49" href="#fn:49">49</a>: 186 <a class="footnote-ref" id="fnref:50" href="#fn:50">50</a> Sometimes a statistical adjustment, such as post-stratification or raking, is used for estimating the selection probability. E.g., when comparing the sample we have with same target population, also known as matching to controls. The estimation process may be focused only on adjusting the existing population to an alternative population (for example, if trying to extrapolate from a panel drawn from several regions to an entire country). In such a case, the adjustment might be focused on some calibration factor 
 
 
 
 
 c
 
 i
 
 
 
 
 {\displaystyle c_{i}}
 
 and the weights be calculated as 
 
 
 
 
 w
 
 i
 
 
 =
 
 
 
 c
 
 i
 
 
 
 
 p
 
 h
 
 
 
 r
 
 h
 
 
 
 
 
 
 
 {\displaystyle w_{i}={\frac {c_{i}}{p_{h}r_{h}}}}
 
.<a class="footnote-ref" id="fnref:51" href="#fn:51">51</a>: 187  However, in other cases, both the under-coverage and non-response are all modeled as part of the statistical adjustment, which leads to an estimation of the overall sampling probability (lets say 
 
 
 
 
 p
 
 i
 
 ′
 
 
 
 {\displaystyle p_{i}'}
 
). In such a case, the weights are simply: 
 
 
 
 
 w
 
 i
 
 
 =
 
 
 1
 
 p
 
 i
 
 ′
 
 
 
 
 
 {\displaystyle w_{i}={\frac {1}{p_{i}'}}}
 
. Notice that when statistical adjustments are used, 
 
 
 
 
 w
 
 i
 
 
 
 
 {\displaystyle w_{i}}
 
 is often estimated based on some model. The formulation in the following sections assume this 
 
 
 
 
 w
 
 i
 
 
 
 
 {\displaystyle w_{i}}
 
 is known, which is not true for statistical adjustments (since we only have 
 
 
 
 
 
 
 
 w
 ^
 
 
 
 
 i
 
 
 
 
 {\displaystyle {\widehat {w}}_{i}}
 
). However, if it is assumed that the estimation error of 
 
 
 
 
 
 
 
 w
 ^
 
 
 
 
 i
 
 
 
 
 {\displaystyle {\widehat {w}}_{i}}
 
 is very small then the following sections can be used as if it was known. Having this assumption be true depends on the size of the sample used for modeling, and is worth keeping in mind during analysis. When the selection probabilities may be different, the sample size is random, and the pairwise selection probabilities are independent, we call this <a href="/facts/Poisson_sampling/3ha7k2y7">Poisson sampling</a>.<a class="footnote-ref" id="fnref:52" href="#fn:52">52</a></li></ol>
<h4>"Design based" vs "model based" for describing properties of estimators</h4>
Adjusting for unequal probability selection through "individual case weights" (e.g. inverse probability weighting), yields various types of estimators for quantities of interest. Estimators such as <a href="/facts/Horvitz%E2%80%93Thompson_estimator/m02k5QFn">Horvitz–Thompson estimator</a> yield unbiased estimators (if the selection probabilities are indeed known, or approximately known), for total and the mean of the population. Deville and Särndal (1992) coined the term "calibration estimator" for estimators using weights such that they satisfy some condition, such as having the sum of weights equal the population size. And more generally, that the weighted sum of weights is equal some quantity of an auxiliary variable: 
 
 
 
 ∑
 
 w
 
 i
 
 
 
 x
 
 i
 
 
 =
 X
 
 
 {\displaystyle \sum w_{i}x_{i}=X}
 
 (e.g., that the sum of weighted ages of the respondents is equal to the population size in each age group).<a class="footnote-ref" id="fnref:53" href="#fn:53">53</a><a class="footnote-ref" id="fnref:54" href="#fn:54">54</a>: 132 <a class="footnote-ref" id="fnref:55" href="#fn:55">55</a>: 1 
The two primary ways to argue about the properties of calibration estimators are:<a class="footnote-ref" id="fnref:56" href="#fn:56">56</a>: 133–134 <a class="footnote-ref" id="fnref:57" href="#fn:57">57</a>

<ol><li>randomization based (or, sampling design based) - in this case, the weights (
 
 
 
 
 w
 
 i
 
 
 
 
 {\displaystyle w_{i}}
 
) and values of the outcome of interest 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 that are measured in the sample are all treated as known. In this framework, there is variability in the (known) values of the outcome (Y). However, the only randomness comes from which of the elements in the population were picked into the sample (often denoted as 
 
 
 
 
 I
 
 i
 
 
 
 
 {\displaystyle I_{i}}
 
, getting 1 if element 
 
 
 
 i
 
 
 {\displaystyle i}
 
 is in the sample and 0 if it is not). For a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>, each 
 
 
 
 
 I
 
 i
 
 
 
 
 {\displaystyle I_{i}}
 
 will be an <a href="/facts/Independent_and_identically_distributed_random_variables/othIRaWt">IID</a> <a href="/facts/Bernoulli_distribution/ChCtYyvs">Bernoulli distribution</a> with some parameter 
 
 
 
 p
 
 
 {\displaystyle p}
 
. For general EPSEM (equal probability sampling) 
 
 
 
 
 I
 
 i
 
 
 
 
 {\displaystyle I_{i}}
 
 will still be Bernoulli with some parameter 
 
 
 
 p
 
 
 {\displaystyle p}
 
, but they may no longer be <a href="/facts/Independence_(probability_theory)/NUzQtnUL">independent</a> random variables. I.e., knowing that a sample is EPSEM means that it maintains marginally equal probability of selection, but it does not inform us about the joint probability of selection. For something like post stratification, the number of elements at each stratum can be modeled as a <a href="/facts/Multinomial_distribution/y58v1p9J">multinomial distribution</a> with different 
 
 
 
 
 p
 
 h
 
 
 
 
 {\displaystyle p_{h}}
 
 inclusion probabilities for each element belonging to some stratum 
 
 
 
 h
 
 
 {\displaystyle h}
 
. In these cases, the sample size itself can be a random variable.</li>
<li>model based - in this case, the sample is fixed, the weights are fixed, but the outcome of interest is treated as a random variable. For example, in the case of post-stratification, the outcome can be modeled as some <a href="/facts/Linear_regression/5n998IhK">linear regression</a> function where the independent variables are indicator variables mapping each observation to its relevant stratum, and the variability comes with the error term.</li></ol>
As we will see later, some proofs in the literature rely on the randomization-based framework, while others focus on the model-based perspective. When moving from the mean to the <a href="/facts/Weighted_mean/VHyJfPxb">weighted mean</a>, more complexity is added. For example, in the context of <a href="/facts/Survey_methodology/vgqyWzAY">survey methodology</a>, often the population size itself is considered an unknown quantity that is estimated. So in the calculation of the weighted mean is in fact based on a <a href="/facts/Ratio_estimator/w2L7MT0L">ratio estimator</a>, with an estimator of the total at the numerator and an estimator of the population size in the denominator (making the <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">variance calculation</a> to be more complex).<a class="footnote-ref" id="fnref:58" href="#fn:58">58</a><a class="footnote-ref" id="fnref:59" href="#fn:59">59</a>: 182 

<h4>Common types of weights</h4>
Table 4: Summary of common types of weights used in design effect calculations<table><tbody><tr><th>Weight Type</th><th>Description</th><th>Interpretation</th></tr><tr><td>Frequency weights</td><td>Each weight is an integer indicating the absolute frequency of an item in the sample</td><td>Specific value has an absolute meaning; weights represent the amount of information in the dataset</td></tr><tr><td>Inverse-variance weights</td><td>Each element is assigned a weight that is the inverse of its known variance</td><td>When all elements have the same expectancy, using such weights for weighted averages has the least variance</td></tr><tr><td>Normalized (convex) weights</td><td>Weights form a convex combination (sum to 1); can be normalized to sum to sample size (n)</td><td>Weights that sum to n have a relative interpretation: elements with weights > 1 are more "rare" than average and have larger influential on (say) the average, while weights < 1 are more "common" and less influential</td></tr><tr><td>Inverse probability weights</td><td>Each element is given a weight proportional to the inverse of its selection probability</td><td>Weights represent how many items each element "represents" in the target population; sum of weights equals the size of the target population</td></tr></tbody></table>
There are many types (and subtypes) of weights, with different ways to use and interpret them. With some weights their absolute value has some important meaning, while with other weights the important part is the relative values of the weights to each other. This section introduces some of the more common types of weights so that they can be referenced in follow-up sections.

<ul><li>Frequency weights<a class="footnote-ref" id="fnref:60" href="#fn:60">60</a> are a basic type of weighting presented in introductory statistics courses. With these, each weight is an integer number that indicates the <a href="/facts/Frequency_(statistics)/FtFNmhDS">absolute frequency</a> of an item in the sample. These are also sometimes termed repeat (or occurrence) weights. The specific value has an absolute meaning that is lost if the weights are transformed, such as when <a href="/facts/Scaling_(geometry)/pFm6fvpu">scaling</a>. For example: if we have the numbers 10 and 20 with the frequency weights values of 2 and 3, then when "spreading" our data it is: 10,10, 20, 20, 20 (with weights of 1 to each of these items). Frequency weights includes the amount of information contained in a dataset, and thus allows things like creating <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">unbiased weighted variance</a> estimation using <a href="/facts/Bessel%27s_correction/VgVzNomA">Bessel's correction</a>. Notice that such weights are often <a href="/facts/Random_variable/TwTBXnLT">random variables</a>, since the specific number of items we will see from each value in the dataset is random.</li>
<li><a href="/facts/Inverse-variance_weighting/1vgxjttI">inverse-variance weighting</a>, also known as analytic weights,<a class="footnote-ref" id="fnref:61" href="#fn:61">61</a> is when each element is assigned a weight that is the inverse of its (known) variance.<a class="footnote-ref" id="fnref:62" href="#fn:62">62</a><a class="footnote-ref" id="fnref:63" href="#fn:63">63</a>: 187  When all elements have the same expectancy, using such weights for calculating <a href="/facts/Weighted_average/VHyJfPxb">weighted averages</a> has the least variance among all weighted averages. In the common formulation, these weights are known and not random.</li>
<li>Normalized (convex) weights is a set of weights that form a <a href="/facts/Convex_combination/YSDu46RI">convex combination</a>, i.e., each weight is a number between 0 and 1, and the sum of all weights is equal to 1. Any set of (non negative) weights can be turned into normalized weights by dividing each weight with the sum of all weights, making these weights normalized to sum to 1.</li></ul>
A related form are weights normalized to sum to sample size (n). These (non-negative) weights sum to the sample size (n), and their mean is 1. Any set of weights can be normalized to sample size by dividing each weight with the average of all weights. These weights have a nice relative interpretation where elements with weights larger than 1 are more "influential" (in terms of their relative influence on, say, the weighted mean) then the average observation, while weights smaller than 1 are less "influential" than the average observation.
<ul><li><a href="/facts/Inverse_probability_weighting/SR1Fy0W7">Inverse probability weighting</a>, or simply probability weights,<a class="footnote-ref" id="fnref:64" href="#fn:64">64</a> is when each element is given a weight that is (proportional) to the inverse probability of selecting that element. E.g., by using 
 
 
 
 
 w
 
 i
 
 
 =
 
 
 1
 
 p
 
 i
 
 
 
 
 
 
 {\displaystyle w_{i}={\frac {1}{p_{i}}}}
 
.<a class="footnote-ref" id="fnref:65" href="#fn:65">65</a>: 185  With inverse probability weights, we learn how many items each element "represents" in the target population. Hence, the sum of such weights returns the size of the target population of interest. Inverse probability weights can be normalized to sum to 1 or normalized to sum to the sample size (n), and many of the calculations from the following sections will yield the same results.</li></ul>
When a sample is <a href="/facts/Simple_random_sample/vDnCHnNx">EPSEM</a> then all the probabilities are equal and the inverse of the selection probability yield weights that are all equal to one another (they are all equal to 
 
 
 
 
 
 N
 n
 
 
 =
 
 
 1
 f
 
 
 
 
 {\displaystyle {\frac {N}{n}}={\frac {1}{f}}}
 
, where 
 
 
 
 n
 
 
 {\displaystyle n}
 
 is the sample size and 
 
 
 
 N
 
 
 {\displaystyle N}
 
 is the population size). Such a sample is called a self weighting sample.<a class="footnote-ref" id="fnref:66" href="#fn:66">66</a>: 193 
There are also indirect ways of applying "weighted" adjustments. For example, the existing cases may be duplicated to <a href="/facts/Imputation_(statistics)/UXBwWUKJ">impute</a> missing observations (e.g. from non-response), with variance estimated using methods such as <a href="/facts/Imputation_(statistics)/UXBwWUKJ">multiple imputation</a>. An alternative approach is to remove (assign a weight of 0 to) some cases. For example, when wanting to reduce the influence of over-sampled groups that are less essential for some analysis. Both cases are similar in nature to inverse probability weighting but the application in practice gives more/less rows of data (making the input potentially simpler to use in some software implementation), instead of applying an extra column of weights. Nevertheless, the consequences of such implementations are similar to just using weights. So while in the case of removing observations the data can easily be handled by common software implementations, the case of adding rows requires special adjustments for the uncertainty estimations. Not doing so may lead to erroneous conclusions(i.e., there is <a href="/facts/No_free_lunch_theorem/JoSR8TTC">no free lunch</a> when using alternative representation of the underlying issues).<a class="footnote-ref" id="fnref:67" href="#fn:67">67</a>: 189, 190 
The term "Haphazard weights", coined by Kish, is used to refer to weights that correspond to unequal selection probabilities, but ones that are not related to the expectancy or variance of the selected elements.<a class="footnote-ref" id="fnref:68" href="#fn:68">68</a>: 190, 191 

<h4>Haphazard weights with estimated ratio-mean - Kish's design effect</h4>
<h5>Formula</h5>
When taking an unrestricted sample of 
 
 
 
 n
 
 
 {\displaystyle n}
 
 elements, we can then randomly split these elements into 
 
 
 
 H
 
 
 {\displaystyle H}
 
 <a href="/facts/Disjoint_sets/nLr8XRVd">disjoint</a> strata, each of them containing some size of 
 
 
 
 
 n
 
 h
 
 
 
 
 {\displaystyle n_{h}}
 
 elements so that 
 
 
 
 
 ∑
 
 h
 =
 1
 
 
 H
 
 
 
 n
 
 h
 
 
 =
 n
 
 
 {\displaystyle \sum \limits _{h=1}^{H}n_{h}=n}
 
. All elements in each stratum 
 
 
 
 h
 
 
 {\displaystyle h}
 
 has some (known) non-negative weight assigned to them (
 
 
 
 
 w
 
 h
 
 
 
 
 {\displaystyle w_{h}}
 
). The weight 
 
 
 
 
 w
 
 h
 
 
 
 
 {\displaystyle w_{h}}
 
 can be produced by the inverse of some unequal selection probability for elements in each stratum 
 
 
 
 h
 
 
 {\displaystyle h}
 
 (i.e., <a href="/facts/Inverse_probability_weighting/SR1Fy0W7">inverse probability weighting</a> following a procedure such as post-stratification). In this setting, Kish's design effect, for the increase in variance of the sample <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">weighted mean</a> due to this design (reflected in the weights), versus <a href="/facts/Simple_random_sample/vDnCHnNx">SRS</a> of some outcome variable y (when there is no correlation between the weights and the outcome, i.e. haphazard weights) is:<a class="footnote-ref" id="fnref:69" href="#fn:69">69</a>: 427 <a class="footnote-ref" id="fnref:70" href="#fn:70">70</a>: 191(4.2)

Deff
        
        =
        
          
            
              n
              
                ∑
                
                  h
                  =
                  1
                
                
                  H
                
              
              (
              
                n
                
                  h
                
              
              
                w
                
                  h
                
                
                  2
                
              
              )
            
            
              (
              
                ∑
                
                  h
                  =
                  1
                
                
                  H
                
              
              
                n
                
                  h
                
              
              
                w
                
                  h
                
              
              
                )
                
                  2
                
              
            
          
        
      
    
    {\displaystyle {\text{Deff}}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{(\sum \limits _{h=1}^{H}n_{h}w_{h})^{2}}}}

By treating each item as coming from its own stratum 
 
 
 
 ∀
 h
 :
 
 n
 
 h
 
 
 =
 1
 
 
 {\displaystyle \forall h:n_{h}=1}
 
, Kish (in 1992) simplified the above formula to the (well-known) following version:<a class="footnote-ref" id="fnref:71" href="#fn:71">71</a>: 191(4.3) <a class="footnote-ref" id="fnref:72" href="#fn:72">72</a>: 318 <a class="footnote-ref" id="fnref:73" href="#fn:73">73</a>: 8

Deff
        
        =
        
          
            
              n
              
                ∑
                
                  i
                  =
                  1
                
                
                  n
                
              
              
                w
                
                  i
                
                
                  2
                
              
            
            
              (
              
                ∑
                
                  i
                  =
                  1
                
                
                  n
                
              
              
                w
                
                  i
                
              
              
                )
                
                  2
                
              
            
          
        
        =
        
          
            
              
                
                  1
                  n
                
              
              
                ∑
                
                  i
                  =
                  1
                
                
                  n
                
              
              
                w
                
                  i
                
                
                  2
                
              
            
            
              
                (
                
                  
                    
                      1
                      n
                    
                  
                  
                    ∑
                    
                      i
                      =
                      1
                    
                    
                      n
                    
                  
                  
                    w
                    
                      i
                    
                  
                
                )
              
              
                2
              
            
          
        
        =
        
          
            
              
                w
                
                  2
                
              
              ¯
            
            
              
                
                  w
                  ¯
                
              
              
                2
              
            
          
        
      
    
    {\displaystyle {\text{Deff}}={\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}={\frac {{\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{2}}{\left({\frac {1}{n}}\sum _{i=1}^{n}w_{i}\right)^{2}}}={\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}}

This version of the formula is valid when one stratum had several observations taken from it (i.e., each having the same weight), or when there are just many strata were each one had one observation taken from it, but several of them had the same probability of selection. While the interpretation is slightly different, the calculation of the two scenarios comes out to be the same.
When using Kish's design effect for unequal weights, you may use the following simplified formula for "<a href="/facts/Leslie_Kish/oy1EYwHC">Kish</a>'s Effective Sample Size"<a class="footnote-ref" id="fnref:74" href="#fn:74">74</a><a class="footnote-ref" id="fnref:75" href="#fn:75">75</a>: 162, 259

n
          
            eff
          
        
        =
        
          
            
              (
              
                ∑
                
                  i
                  =
                  1
                
                
                  n
                
              
              
                w
                
                  i
                
              
              
                )
                
                  2
                
              
            
            
              
                ∑
                
                  i
                  =
                  1
                
                
                  n
                
              
              
                w
                
                  i
                
                
                  2
                
              
            
          
        
      
    
    {\displaystyle n_{\text{eff}}={\frac {(\sum _{i=1}^{n}w_{i})^{2}}{\sum _{i=1}^{n}w_{i}^{2}}}}

<h5>Assumptions and proofs</h5>
The above formula, by <a href="/facts/Leslie_Kish/oy1EYwHC">Kish</a>, gives the increase in the variance of the <a href="/facts/Weighted_mean/VHyJfPxb">weighted mean</a> based on "haphazard" weights. This can also be written as the following formula where y are observations selected using unequal selection probabilities (with no within-cluster correlation, and no relationship to the expectancy or variance of the outcome measurement),<a class="footnote-ref" id="fnref:76" href="#fn:76">76</a>: 190, 191  and y' are the observations we would have had if we got them from a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>:

 
 
 
 
 
 Deff
 
 
 Kish
 
 
 =
 
 
 
 
 var
 
 
 (
 
 
 
 
 y
 ¯
 
 
 
 
 w
 
 
 )
 
 
 
 
 var
 
 
 (
 
 
 
 
 y
 ¯
 
 
 
 ′
 
 )
 
 
 
 
 =
 
 
 
 
 var
 
 
 (
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 w
 
 i
 
 
 
 y
 
 i
 
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 w
 
 i
 
 
 
 
 
 )
 
 
 
 
 var
 
 
 (
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 n
 
 
 
 y
 
 i
 
 ′
 
 
 n
 
 
 )
 
 
 
 
 
 
 {\displaystyle {\text{Deff}}_{\text{Kish}}={\frac {{\text{var}}\left({\bar {y}}_{w}\right)}{{\text{var}}\left({\bar {y}}'\right)}}={\frac {{\text{var}}\left({\frac {\sum \limits _{i=1}^{n}w_{i}y_{i}}{\sum \limits _{i=1}^{n}w_{i}}}\right)}{{\text{var}}\left({\frac {\sum \limits _{i=1}^{n}y_{i}'}{n}}\right)}}}

It can be shown that the ratio of variances formula can be reduced to Kish's formula by using a model based perspective.<a class="footnote-ref" id="fnref:77" href="#fn:77">77</a> In it, Kish's formula will hold when all n observations (
 
 
 
 
 y
 
 1
 
 
 ,
 .
 .
 .
 ,
 
 y
 
 n
 
 
 
 
 {\displaystyle y_{1},...,y_{n}}
 
) are (at least approximately) <a href="/facts/Uncorrelatedness_(probability_theory)/Ub2XK1WJ">uncorrelated</a> (
 
 
 
 ∀
 (
 i
 ≠
 j
 )
 :
 
 cor
 
 (
 
 y
 
 i
 
 
 ,
 
 y
 
 j
 
 
 )
 =
 0
 
 
 {\displaystyle \forall (i\neq j):{\text{cor}}(y_{i},y_{j})=0}
 
), with the same <a href="/facts/Variance/ULBJKXD1">variance</a> (
 
 
 
 
 σ
 
 2
 
 
 
 
 {\displaystyle \sigma ^{2}}
 
) in the response variable of interest (y). It will also be required to assume the weights themselves are not a <a href="/facts/Random_variable/TwTBXnLT">random variable</a> but rather some known constants (e.g. the inverse of probability of selection, for some pre-determined and known <a href="/facts/Sampling_(statistics)/kIb01xdL">sampling design</a>).

The conditions on y are trivially held if the y observations are <a href="/facts/Independent_and_identically_distributed_random_variables/othIRaWt">IID</a> with the same <a href="/facts/Expected_value/1XV0JKL8">expectation</a> and <a href="/facts/Variance/ULBJKXD1">variance</a>. In such cases, 
 
 
 
 y
 =
 
 y
 ′
 
 
 
 {\displaystyle y=y'}
 
, and we can estimate 
 
 
 
 v
 a
 r
 
 (
 
 
 
 
 y
 ¯
 
 
 
 
 w
 
 
 )
 
 
 
 {\displaystyle var\left({\bar {y}}_{w}\right)}
 
 by using 
 
 
 
 
 
 
 
 var
 
 
 (
 
 
 
 
 y
 ¯
 
 
 
 
 w
 
 
 )
 
 
 ¯
 
 
 =
 
 
 
 
 var
 
 
 (
 
 
 
 y
 ¯
 
 
 
 )
 
 
 ¯
 
 
 ×
 
 Deff
 
 
 
 {\displaystyle {\overline {{\text{var}}\left({\bar {y}}_{w}\right)}}={\overline {{\text{var}}\left({\bar {y}}\right)}}\times {\text{Deff}}}
 
.<a class="footnote-ref" id="fnref:78" href="#fn:78">78</a><a class="footnote-ref" id="fnref:79" href="#fn:79">79</a> If the y's are not all with the same expectations then we cannot use the estimated variance for calculation, since that estimation assumes that all 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
s have the same expectation. Specifically, if there is a correlation between the weights and the outcome variable y, then it means that the expectation of y is not the same for all observations (but rather, dependent on the specific weight value for each observation). In such a case, while the design effect formula might still be correct (if the other conditions are met), it would require a different estimator for the variance of the weighted mean. For example, it might be better to use a <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">weighted variance estimator</a>.
If different 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
s values have different variances, then while the weighted variance could capture the correct population-level variance, Kish's formula for the design effect may no longer be true.
A similar issue happens if there is some correlation structure in the samples (such as when using <a href="/facts/Cluster_sampling/2PX8m0fd">cluster sampling</a>).

<h5>Relation to the coefficient of variation</h5>
Notice that Kish's definition of the design effect is closely tied to the <a href="/facts/Coefficient_of_variation/V8ddKkz4">coefficient of variation</a> (Kish also calls it relvariance or relvar for short<a class="footnote-ref" id="fnref:80" href="#fn:80">80</a>) of the weights (when using the <a href="/facts/Standard_deviation/qSug9BRl">uncorrected (population level) sample standard deviation</a> for <a href="/facts/Coefficient_of_variation/V8ddKkz4">estimation</a>). This has several notations in the literature:<a class="footnote-ref" id="fnref:81" href="#fn:81">81</a>: 191 <a class="footnote-ref" id="fnref:82" href="#fn:82">82</a>: 396

Deff
 
 =
 1
 +
 L
 =
 1
 +
 
 
 
 C
 
 V
 
 
 
 
 2
 
 
 =
 1
 +
 
 relvar
 
 (
 w
 )
 =
 1
 +
 
 
 
 V
 (
 w
 )
 
 
 
 
 
 w
 ¯
 
 
 
 
 2
 
 
 
 
 
 
 {\displaystyle {\text{Deff}}=1+L=1+{C_{V}}^{2}=1+{\text{relvar}}(w)=1+{\frac {V(w)}{{\bar {w}}^{2}}}}
 
.
Where 
 
 
 
 V
 (
 w
 )
 =
 
 
 
 ∑
 (
 
 w
 
 i
 
 
 −
 
 
 
 w
 ¯
 
 
 
 
 )
 
 2
 
 
 
 n
 
 
 
 
 {\displaystyle V(w)={\frac {\sum (w_{i}-{\bar {w}})^{2}}{n}}}
 
 is the population variance of 
 
 
 
 w
 
 
 {\displaystyle w}
 
, and 
 
 
 
 
 
 
 w
 ¯
 
 
 
 =
 
 
 
 ∑
 
 w
 
 i
 
 
 
 n
 
 
 
 
 {\displaystyle {\bar {w}}={\frac {\sum w_{i}}{n}}}
 
 is the mean. When the weights are normalized to sample size (so that their sum is equal to n and their mean is equal to 1), then 
 
 
 
 
 
 
 C
 
 V
 
 
 
 
 2
 
 
 =
 V
 (
 w
 )
 
 
 {\displaystyle {C_{V}}^{2}=V(w)}
 
 and the formula reduces to 
 
 
 
 
 Deff
 
 =
 1
 +
 V
 (
 w
 )
 
 
 {\displaystyle {\text{Deff}}=1+V(w)}
 
. While it is true we assume the weights are fixed, we can think of their variance as the variance of an <a href="/facts/Empirical_distribution_function/frWIF9F9">empirical distribution</a> defined by sampling (with equal probability) one weight from our set of weights (similar to how we would think about the correlation of x and y in a <a href="/facts/Simple_linear_regression/SNINCEFU">simple linear regression</a>).

<h5>Relation to disproportionate stratified sampling</h5>
Kish's original definition compared the variance under some sampling design to the variance achieved through a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>. Some literature provide the following alternative definition for Kish's design effect: "the ratio of the variance of the weighted survey mean under disproportionate stratified sampling to the variance under <a href="/facts/Stratified_sampling/CvAUCegT">proportionate stratified sampling</a> when all stratum unit variances are equal".<a class="footnote-ref" id="fnref:83" href="#fn:83">83</a>: 318 <a class="footnote-ref" id="fnref:84" href="#fn:84">84</a>: 396  Reflecting on this, Park and Lee (2006) stated that "The rationale behind [...][Kish's] derivation is that the loss in precision of [the weighted mean] due to haphazard unequal weighting can be approximated by the ratio of the variance under disproportionate stratified sampling to that under the proportionate stratified sampling".<a class="footnote-ref" id="fnref:85" href="#fn:85">85</a>: 8 
Note that this alternative definition only approximated since if the denominator is based on "proportionate stratified sampling" (achieved via <a href="/facts/Stratified_sampling/CvAUCegT">stratified sampling</a>) then such a selection will yield a reduced variance as compared with <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>. This is since stratified sampling removes some of the variability in the specific number of elements per stratum, as occurs under SRS.
Relatedly, Cochran (1977) provides a formula for the proportional increase in variance due to deviation from optimum allocation (what, in Kish's formulas, would be called L).<a class="footnote-ref" id="fnref:86" href="#fn:86">86</a>: 116 

<h5>Alternative naming conventions</h5>
Early papers used the term 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
.<a class="footnote-ref" id="fnref:87" href="#fn:87">87</a>: 192  As more definitions of the design effect appeared, Kish's design effect for unequal selection probabilities was denoted 
 
 
 
 
 
 Deff
 
 
 Kish
 
 
 
 
 {\displaystyle {\text{Deff}}_{\text{Kish}}}
 
 (or 
 
 
 
 
 
 Deft
 
 
 Kish
 
 
 2
 
 
 
 
 {\displaystyle {\text{Deft}}_{\text{Kish}}^{2}}
 
) or simply 
 
 
 
 
 
 Deff
 
 
 K
 
 
 
 
 {\displaystyle {\text{Deff}}_{K}}
 
 for short.<a class="footnote-ref" id="fnref:88" href="#fn:88">88</a>: 8 <a class="footnote-ref" id="fnref:89" href="#fn:89">89</a>: 396 <a class="footnote-ref" id="fnref:90" href="#fn:90">90</a>: 318  Kish's design effect is also known as the "Unequal Weighting Effect" (or just UWE), termed by Liu et al. in 2002.<a class="footnote-ref" id="fnref:91" href="#fn:91">91</a>: 2124 

<h4>When the outcome correlates with the selection probabilities</h4>
<h5>Spencer's Deff for estimated total</h5>
The estimator for the total is the "p-expanded with replacement" estimator (a.k.a.: pwr-estimator or <a href="/facts/Horvitz%E2%80%93Thompson_estimator/m02k5QFn">Hansen and Hurwitz</a>). It is based on a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a> (with replacement, denoted SIR) of n items (
 
 
 
 
 y
 
 k
 
 
 
 
 {\displaystyle y_{k}}
 
) from a population of size N.<a class="footnote-ref" id="fnref:92" href="#fn:92">92</a> Each item has a probability of 
 
 
 
 
 p
 
 k
 
 
 
 
 {\displaystyle p_{k}}
 
 (k from 1 to N) to be drawn in a single draw (
 
 
 
 
 ∑
 
 U
 
 
 
 p
 
 k
 
 
 =
 1
 
 
 {\displaystyle \sum _{U}p_{k}=1}
 
, i.e. it is a <a href="/facts/Multinomial_distribution/y58v1p9J">multinomial distribution</a>). The probability that a specific 
 
 
 
 
 y
 
 k
 
 
 
 
 {\displaystyle y_{k}}
 
 will appear in the sample is 
 
 
 
 
 p
 
 k
 
 
 
 
 {\displaystyle p_{k}}
 
. The "p-expanded with replacement" value is 
 
 
 
 
 Z
 
 i
 
 
 =
 
 
 
 y
 
 k
 
 
 
 p
 
 k
 
 
 
 
 
 
 {\displaystyle Z_{i}={\frac {y_{k}}{p_{k}}}}
 
 with the following expectancy: 
 
 
 
 E
 [
 
 Z
 
 i
 
 
 ]
 =
 E
 [
 
 I
 
 i
 
 
 
 
 
 y
 
 k
 
 
 
 p
 
 k
 
 
 
 
 ]
 =
 
 
 
 y
 
 k
 
 
 
 p
 
 k
 
 
 
 
 E
 [
 
 I
 
 i
 
 
 ]
 =
 
 
 
 y
 
 k
 
 
 
 p
 
 k
 
 
 
 
 
 p
 
 k
 
 
 =
 
 y
 
 k
 
 
 
 
 {\displaystyle E[Z_{i}]=E[I_{i}{\frac {y_{k}}{p_{k}}}]={\frac {y_{k}}{p_{k}}}E[I_{i}]={\frac {y_{k}}{p_{k}}}p_{k}=y_{k}}
 
. Hence 
 
 
 
 
 
 
 
 Y
 ^
 
 
 
 
 p
 w
 r
 
 
 =
 
 
 1
 n
 
 
 
 ∑
 
 i
 
 
 n
 
 
 
 Z
 
 i
 
 
 
 
 {\displaystyle {\hat {Y}}_{pwr}={\frac {1}{n}}\sum _{i}^{n}Z_{i}}
 
, the pwr-estimator, is an unbiased estimator for the sum total of y.<a class="footnote-ref" id="fnref:93" href="#fn:93">93</a>: 51 
In 2000, Bruce D. Spencer proposed a formula for estimating the design effect for the variance of estimating the total (not the mean) of some quantity (
 
 
 
 
 
 
 Y
 ^
 
 
 
 
 
 {\displaystyle {\hat {Y}}}
 
), when there is correlation between the selection probabilities of the elements and the outcome variable of interest.<a class="footnote-ref" id="fnref:94" href="#fn:94">94</a>
In this setup, a sample of size n is drawn (with replacement) from a population of size N. Each item is drawn with probability 
 
 
 
 
 P
 
 i
 
 
 
 
 {\displaystyle P_{i}}
 
 (where 
 
 
 
 
 ∑
 
 i
 =
 1
 
 
 N
 
 
 
 P
 
 i
 
 
 =
 1
 
 
 {\displaystyle \sum _{i=1}^{N}P_{i}=1}
 
, i.e. <a href="/facts/Multinomial_distribution/y58v1p9J">multinomial distribution</a>). The selection probabilities are used to define the Normalized (convex) weights: 
 
 
 
 
 w
 
 i
 
 
 =
 
 
 1
 
 n
 
 P
 
 i
 
 
 
 
 
 
 
 {\displaystyle w_{i}={\frac {1}{nP_{i}}}}
 
. Notice that for some random set of n items, the sum of weights will be equal to 1 only by expectation (
 
 
 
 E
 [
 
 w
 
 i
 
 
 ]
 =
 1
 
 
 {\displaystyle E[w_{i}]=1}
 
) with some variability of the sum around it (i.e., the sum of elements from a <a href="/facts/Poisson_binomial_distribution/KnPnMWtN">Poisson binomial distribution</a>). The relationship between 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 and 
 
 
 
 
 P
 
 i
 
 
 
 
 {\displaystyle P_{i}}
 
 is defined by the following (population) <a href="/facts/Simple_linear_regression/SNINCEFU">simple linear regression</a>:

y
          
            i
          
        
        =
        α
        +
        β
        
          P
          
            i
          
        
        +
        
          ϵ
          
            i
          
        
      
    
    {\displaystyle y_{i}=\alpha +\beta P_{i}+\epsilon _{i}}

Where 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 is the outcome of element i, which linearly depends on 
 
 
 
 
 P
 
 i
 
 
 
 
 {\displaystyle P_{i}}
 
 with the intercept 
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
 and slope 
 
 
 
 β
 
 
 {\displaystyle \beta }
 
. The residual from the fitted line is 
 
 
 
 
 ϵ
 
 i
 
 
 =
 
 y
 
 i
 
 
 −
 (
 α
 +
 β
 
 P
 
 i
 
 
 )
 
 
 {\displaystyle \epsilon _{i}=y_{i}-(\alpha +\beta P_{i})}
 
. We can also define the population variances of the outcome and the residuals as 
 
 
 
 
 σ
 
 y
 
 
 2
 
 
 
 
 {\displaystyle \sigma _{y}^{2}}
 
 and 
 
 
 
 
 σ
 
 ϵ
 
 
 2
 
 
 
 
 {\displaystyle \sigma _{\epsilon }^{2}}
 
. The correlation between 
 
 
 
 
 P
 
 i
 
 
 
 
 {\displaystyle P_{i}}
 
 and 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 is 
 
 
 
 
 ρ
 
 y
 ,
 P
 
 
 
 
 {\displaystyle \rho _{y,P}}
 
.
Spencer's (approximate) design effect for estimating the total of y is:<a class="footnote-ref" id="fnref:95" href="#fn:95">95</a>: 138 <a class="footnote-ref" id="fnref:96" href="#fn:96">96</a>: 4 <a class="footnote-ref" id="fnref:97" href="#fn:97">97</a>: 401

Deff
          
          
            S
            p
            e
            n
            c
            e
            r
          
        
        =
        (
        1
        −
        
          
            
              
                ρ
                ^
              
            
          
          
            y
            ,
            P
          
          
            2
          
        
        )
        (
        1
        +
        L
        )
        +
        
          
            (
            
              
                
                  
                    α
                    ^
                  
                
                
                  
                    
                      
                        σ
                        ^
                      
                    
                  
                  
                    y
                  
                
              
            
            )
          
          
            2
          
        
        L
      
    
    {\displaystyle {\text{Deff}}_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+L)+\left({\frac {\hat {\alpha }}{{\hat {\sigma }}_{y}}}\right)^{2}L}

Where:

<ul><li>
 
 
 
 
 
 
 
 ρ
 ^
 
 
 
 
 y
 ,
 P
 
 
 2
 
 
 
 
 {\displaystyle {\hat {\rho }}_{y,P}^{2}}
 
 estimates 
 
 
 
 
 ρ
 
 y
 ,
 P
 
 
 2
 
 
 
 
 {\displaystyle \rho _{y,P}^{2}}
 
</li>
<li>
 
 
 
 
 
 
 α
 ^
 
 
 
 
 
 {\displaystyle {\hat {\alpha }}}
 
 estimates the slope 
 
 
 
 α
 
 
 {\displaystyle \alpha }
 
</li>
<li>
 
 
 
 
 
 
 
 σ
 ^
 
 
 
 
 y
 
 
 
 
 {\displaystyle {\hat {\sigma }}_{y}}
 
 estimates the population variance 
 
 
 
 
 σ
 
 y
 
 
 
 
 {\displaystyle \sigma _{y}}
 
, and</li>
<li>L is the relvariance of the weights, as defined in Kish's formula: 
 
 
 
 L
 =
 c
 
 v
 
 w
 
 
 2
 
 
 =
 
 relvar
 
 (
 w
 )
 =
 
 
 
 V
 (
 w
 )
 
 
 
 
 
 w
 ¯
 
 
 
 
 2
 
 
 
 
 
 
 {\displaystyle L=cv_{w}^{2}={\text{relvar}}(w)={\frac {V(w)}{{\bar {w}}^{2}}}}
 
.</li></ul>
This assumes that the regression model fits well so that the probability of selection and the residuals are <a href="/facts/Independence_(probability_theory)/NUzQtnUL">independent</a>, since it leads to the residuals, and the square residuals, to be uncorrelated with the weights, i.e., that 
 
 
 
 
 ρ
 
 ϵ
 ,
 W
 
 
 =
 0
 
 
 {\displaystyle \rho _{\epsilon ,W}=0}
 
 and also 
 
 
 
 
 ρ
 
 
 ϵ
 
 2
 
 
 ,
 W
 
 
 =
 0
 
 
 {\displaystyle \rho _{\epsilon ^{2},W}=0}
 
.<a class="footnote-ref" id="fnref:98" href="#fn:98">98</a>: 138 
When the population size (N) is very large, the formula can be written as:<a class="footnote-ref" id="fnref:99" href="#fn:99">99</a>: 319

Deff
          
          
            S
            p
            e
            n
            c
            e
            r
          
        
        =
        (
        1
        −
        
          
            
              
                ρ
                ^
              
            
          
          
            y
            ,
            P
          
          
            2
          
        
        )
        (
        1
        +
        c
        
          v
          
            w
          
          
            2
          
        
        )
        +
        
          
            (
            
              
                1
                
                  c
                  
                    v
                    
                      Y
                    
                    
                      2
                    
                  
                
              
            
            )
          
          
            2
          
        
        c
        
          v
          
            w
          
          
            2
          
        
      
    
    {\displaystyle {\text{Deff}}_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+\left({\frac {1}{cv_{Y}^{2}}}\right)^{2}cv_{w}^{2}}

(since 
 
 
 
 α
 =
 
 
 
 Y
 ¯
 
 
 
 −
 β
 ×
 
 
 
 P
 ¯
 
 
 
 =
 
 
 
 Y
 ¯
 
 
 
 −
 β
 ×
 
 
 1
 N
 
 
 ≈
 
 
 
 Y
 ¯
 
 
 
 
 
 {\displaystyle \alpha ={\bar {Y}}-\beta \times {\bar {P}}={\bar {Y}}-\beta \times {\frac {1}{N}}\approx {\bar {Y}}}
 
, where 
 
 
 
 c
 
 v
 
 Y
 
 
 2
 
 
 =
 
 
 
 σ
 
 Y
 
 
 2
 
 
 
 
 
 
 Y
 ¯
 
 
 
 
 2
 
 
 
 
 
 
 {\displaystyle cv_{Y}^{2}={\frac {\sigma _{Y}^{2}}{{\bar {Y}}^{2}}}}
 
)
This approximation assumes that the linear relationship between P and y holds. And also that the correlation of the weights with the errors, and the errors squared, are both zero. I.e., 
 
 
 
 
 ρ
 
 w
 ,
 e
 
 
 =
 0
 
 
 {\displaystyle \rho _{w,e}=0}
 
 and 
 
 
 
 
 ρ
 
 w
 ,
 
 e
 
 2
 
 
 
 
 =
 0
 
 
 {\displaystyle \rho _{w,e^{2}}=0}
 
.<a class="footnote-ref" id="fnref:100" href="#fn:100">100</a>: 4 
We notice that if 
 
 
 
 
 
 
 
 ρ
 ^
 
 
 
 
 y
 ,
 P
 
 
 ≈
 0
 
 
 {\displaystyle {\hat {\rho }}_{y,P}\approx 0}
 
, then 
 
 
 
 
 
 
 α
 ^
 
 
 
 ≈
 
 
 
 y
 ¯
 
 
 
 
 
 {\displaystyle {\hat {\alpha }}\approx {\bar {y}}}
 
 (i.e., the average of y). In such a case, the formula reduces to

Deff
          
          
            S
            p
            e
            n
            c
            e
            r
          
        
        =
        (
        1
        +
        L
        )
        +
        
          
            (
            
              
                1
                
                  
                    relvar
                  
                  (
                  y
                  )
                
              
            
            )
          
          
            2
          
        
        L
      
    
    {\displaystyle {\text{Deff}}_{Spencer}=(1+L)+\left({\frac {1}{{\text{relvar}}(y)}}\right)^{2}L}

Only if the variance of y is much larger than its mean, then the right-most term is close to 0 (i.e., 
 
 
 
 
 
 1
 
 
 relvar
 
 (
 y
 )
 
 
 
 =
 
 
 
 
 Y
 ¯
 
 
 
 σ
 
 y
 
 
 
 
 ≈
 0
 
 
 {\displaystyle {\frac {1}{{\text{relvar}}(y)}}={\frac {\bar {Y}}{\sigma _{y}}}\approx 0}
 
), which reduces Spencer's design effect (for the estimated total) to be equal to Kish's design effect (for the ratio means):<a class="footnote-ref" id="fnref:101" href="#fn:101">101</a>: 5  
 
 
 
 
 
 Deff
 
 
 S
 p
 e
 n
 c
 e
 r
 
 
 ≈
 (
 1
 +
 L
 )
 =
 
 
 Deff
 
 
 Kish
 
 
 
 
 {\displaystyle {\text{Deff}}_{Spencer}\approx (1+L)={\text{Deff}}_{\text{Kish}}}
 
. Otherwise, the two formulas will yield different results, which demonstrates the difference between the design effect of the total vs. the design effect of the mean.

<h5>Park and Lee's Deff for estimated ratio-mean</h5>
In 2001, Park and Lee extended Spencer's formula to the case of the ratio-mean (i.e., estimating the mean by dividing the estimator of the total with the estimator of the population size). It is:<a class="footnote-ref" id="fnref:102" href="#fn:102">102</a>: 4

Deff
          
          
            P
            a
            r
            k
            &
            L
            e
            e
          
        
        =
        (
        1
        −
        
          
            
              
                ρ
                ^
              
            
          
          
            y
            ,
            P
          
          
            2
          
        
        )
        (
        1
        +
        c
        
          v
          
            w
          
          
            2
          
        
        )
        +
        
          
            
              
                
                  
                    ρ
                    ^
                  
                
              
              
                y
                ,
                P
              
              
                2
              
            
            
              c
              
                v
                
                  P
                
                
                  2
                
              
            
          
        
        c
        
          v
          
            w
          
          
            2
          
        
      
    
    {\displaystyle {\text{Deff}}_{Park\&Lee}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+{\frac {{\hat {\rho }}_{y,P}^{2}}{cv_{P}^{2}}}cv_{w}^{2}}

Where:

<ul><li>
 
 
 
 c
 
 v
 
 P
 
 
 2
 
 
 
 
 {\displaystyle cv_{P}^{2}}
 
 is the (estimated) squared coefficient of variation of the probabilities of selection.</li></ul>
Park and Lee's formula is exactly equal to Kish's formula when 
 
 
 
 
 
 
 
 ρ
 ^
 
 
 
 
 y
 ,
 P
 
 
 2
 
 
 =
 0
 
 
 {\displaystyle {\hat {\rho }}_{y,P}^{2}=0}
 
. Both formulas relate to the design effect of the mean of y, while Spencer's 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 relates to the estimation of the population total.
In general, the 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 for the total (
 
 
 
 
 
 
 Y
 ^
 
 
 
 
 
 {\displaystyle {\hat {Y}}}
 
) tends to be less efficient than the 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 for the ratio mean (
 
 
 
 
 
 
 
 
 Y
 ¯
 
 
 ^
 
 
 
 
 
 {\displaystyle {\hat {\bar {Y}}}}
 
) when 
 
 
 
 
 ρ
 
 y
 ,
 P
 
 
 
 
 {\displaystyle \rho _{y,P}}
 
 is small. And in general, 
 
 
 
 
 ρ
 
 y
 ,
 P
 
 
 
 
 {\displaystyle \rho _{y,P}}
 
 impacts the efficiency of both design effects.<a class="footnote-ref" id="fnref:103" href="#fn:103">103</a>: 8 

<h3>Cluster sampling</h3>
For data collected using <a href="/facts/Cluster_sampling/2PX8m0fd">cluster sampling</a> we assume the following structure:

<ul><li>
 
 
 
 
 n
 
 k
 
 
 
 
 {\displaystyle n_{k}}
 
 observations in each cluster and K clusters, and with a total of 
 
 
 
 n
 =
 ∑
 
 n
 
 k
 
 
 
 
 {\displaystyle n=\sum n_{k}}
 
 observations.</li>
<li>The observations have a <a href="/facts/Block_matrix/fPI8HX9f">block diagonal</a> correlation matrix in which every pair of observations from the same cluster is correlated with an <a href="/facts/Intraclass_correlation/GjhrGfUD">intra-class correlation</a> of 
 
 
 
 ρ
 
 
 {\displaystyle \rho }
 
, while every pair from difference clusters are uncorrelated.<a class="footnote-ref" id="fnref:104" href="#fn:104">104</a> I.e., for every pair of observations, 
 
 
 
 i
 
 
 {\displaystyle i}
 
 and 
 
 
 
 j
 
 
 {\displaystyle j}
 
, if they belong to the same cluster 
 
 
 
 k
 
 
 {\displaystyle k}
 
, we get 
 
 
 
 c
 o
 v
 (
 
 y
 
 i
 
 
 ,
 
 y
 
 j
 
 
 )
 =
 ρ
 
 σ
 
 2
 
 
 
 
 {\displaystyle cov(y_{i},y_{j})=\rho \sigma ^{2}}
 
. And two items from two different clusters are not correlated, i.e.i.e.: 
 
 
 
 c
 o
 v
 (
 
 y
 
 i
 
 
 ,
 
 y
 
 j
 
 
 )
 =
 0
 
 
 {\displaystyle cov(y_{i},y_{j})=0}
 
.</li>
<li>An element from any cluster is assumed to have the same variance: 
 
 
 
 
 var
 
 (
 
 y
 
 i
 
 
 )
 =
 
 σ
 
 h
 
 
 2
 
 
 =
 
 σ
 
 2
 
 
 
 
 {\displaystyle {\text{var}}(y_{i})=\sigma _{h}^{2}=\sigma ^{2}}
 
.</li></ul>
When clusters are all of the same size 
 
 
 
 
 n
 
 ∗
 
 
 
 
 {\displaystyle n^{*}}
 
, the design effect Deff, proposed by Kish in 1965 (and later re-visited by others), is given by:<a class="footnote-ref" id="fnref:105" href="#fn:105">105</a>: 162 <a class="footnote-ref" id="fnref:106" href="#fn:106">106</a>: 399 <a class="footnote-ref" id="fnref:107" href="#fn:107">107</a>: 9 <a class="footnote-ref" id="fnref:108" href="#fn:108">108</a><a class="footnote-ref" id="fnref:109" href="#fn:109">109</a><a class="footnote-ref" id="fnref:110" href="#fn:110">110</a>: 241

Deff
        
        =
        1
        +
        (
        
          n
          
            ∗
          
        
        −
        1
        )
        ρ
        .
      
    
    {\displaystyle {\text{Deff}}=1+(n^{*}-1)\rho .}

It is sometimes also denoted as 
 
 
 
 
 
 Deff
 
 
 C
 
 
 
 
 {\displaystyle {\text{Deff}}_{C}}
 
.<a class="footnote-ref" id="fnref:111" href="#fn:111">111</a>: 2124 
In various papers, when cluster sizes are not equal, the above formula is also used with 
 
 
 
 
 n
 
 ∗
 
 
 
 
 {\displaystyle n^{*}}
 
 as the average cluster size (which is also sometimes denoted as 
 
 
 
 
 
 
 b
 ¯
 
 
 
 
 
 {\displaystyle {\bar {b}}}
 
).<a class="footnote-ref" id="fnref:112" href="#fn:112">112</a><a class="footnote-ref" id="fnref:113" href="#fn:113">113</a>: 105  In such cases, Kish's formula (using the average cluster weight) serves as a conservative (upper bound) of the exact design effect.<a class="footnote-ref" id="fnref:114" href="#fn:114">114</a>: 106 
Alternative formulas exists for unequal cluster sizes.<a class="footnote-ref" id="fnref:115" href="#fn:115">115</a>: 193  Followup work had discussed the sensitivity of using the average cluster size with various assumptions.<a class="footnote-ref" id="fnref:116" href="#fn:116">116</a>

<h3>The design effect for complex designs</h3>
<h4>Unequal selection probabilities times Cluster sampling</h4>
In a 1987 paper, Kish proposed a combined design effect that incorporates both the effects due to weighting that accounts for unequal selection probabilities and cluster sampling:<a class="footnote-ref" id="fnref:117" href="#fn:117">117</a>: 16 <a class="footnote-ref" id="fnref:118" href="#fn:118">118</a>: 105 <a class="footnote-ref" id="fnref:119" href="#fn:119">119</a>: 4 <a class="footnote-ref" id="fnref:120" href="#fn:120">120</a>: 2

Deff
          
          
            Kish
          
        
        =
        
          
            
              n
              
                ∑
                
                  h
                  =
                  1
                
                
                  H
                
              
              (
              
                n
                
                  h
                
              
              
                w
                
                  h
                
                
                  2
                
              
              )
            
            
              
                (
                
                  
                    ∑
                    
                      h
                      =
                      1
                    
                    
                      H
                    
                  
                  
                    n
                    
                      h
                    
                  
                  
                    w
                    
                      h
                    
                  
                
                )
              
              
                2
              
            
          
        
        
          (
          
            1
            +
            (
            
              n
              
                ∗
              
            
            −
            1
            )
            ρ
          
          )
        
        =
        
          
            Deff
          
          
            k
          
        
        ×
        
          
            Deff
          
          
            C
          
        
      
    
    {\displaystyle {\text{Deff}}_{\text{Kish}}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{\left(\sum \limits _{h=1}^{H}n_{h}w_{h}\right)^{2}}}\left(1+(n^{*}-1)\rho \right)={\text{Deff}}_{k}\times {\text{Deff}}_{C}}

The above uses notations similar to what is used in this article (the original 1987 publication used different notation).<a class="footnote-ref" id="fnref:121" href="#fn:121">121</a> A model based justification for this formula was provided by Gabler et al.<a class="footnote-ref" id="fnref:122" href="#fn:122">122</a>

<h4>Stratified sampling times unequal selection probabilities times Cluster sampling</h4>
In 2000, Liu and Aragon proposed a decomposition of unequal selection probabilities design effect for different strata in stratified sampling.<a class="footnote-ref" id="fnref:123" href="#fn:123">123</a> In 2002, Liu et al. extended that work to account for stratified samples, where within each stratum is a set of unequal selection probability weights. The cluster sampling is either global or per stratum.<a class="footnote-ref" id="fnref:124" href="#fn:124">124</a> Similar work was done also by Park et al. in 2003.<a class="footnote-ref" id="fnref:125" href="#fn:125">125</a>

<h4>Chen-Rust Deff: Design effects to two- and three-stage designs with stratification</h4>
The Chen-Rust 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
 extends the model-based justification of Kish’s 1987 formula for design effects proposed by Gabler, el. al.,<a class="footnote-ref" id="fnref:126" href="#fn:126">126</a> applying it to two-stage designs with stratification at the first stage and to three-stage designs without stratification.<a class="footnote-ref" id="fnref:127" href="#fn:127">127</a> The modified formulae define the overall design effect using survey weights and population intracluster correlations. These formulae allow for insightful interpretations of design effects from various sources and can estimate intracluster correlations in completed surveys or predict design effects in future surveys.

<h4>Henry's Deff: a design effect measure for calibration weighting in single-stage samples</h4>
Henry's 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
<a class="footnote-ref" id="fnref:128" href="#fn:128">128</a> proposes an extended model-assisted weighting design-effect measure for single-stage sampling and calibration weight adjustments for a case where 
 
 
 
 
 y
 
 i
 
 
 =
 α
 +
 β
 
 x
 
 i
 
 
 +
 
 ϵ
 
 i
 
 
 
 
 {\displaystyle y_{i}=\alpha +\beta x_{i}+\epsilon _{i}}
 
, where 
 
 
 
 
 x
 
 i
 
 
 
 
 {\displaystyle x_{i}}
 
 is a vector of covariates, the model errors are independent, and the estimator of the population total is the general regression estimator (GREG) of Särndal, Swensson, and Wretman (1992).<a class="footnote-ref" id="fnref:129" href="#fn:129">129</a> The new measure considers the combined effects of non-epsem sampling design, unequal weights from calibration adjustments, and the correlation between an analysis variable and the auxiliaries used in calibration.

<h4>Lohr's Deff: a design effect for a regression slope in a cluster sample</h4>
Lohr's 
 
 
 
 
 Deff
 
 
 
 {\displaystyle {\text{Deff}}}
 
<a class="footnote-ref" id="fnref:130" href="#fn:130">130</a> is for ordinary least squares (OLS) and generalized least squares (GLS) estimators in the context of cluster sampling, using a random coefficient regression model. Lohr presents conditions under which the GLS estimator of the regression slope has a design effect less than 1, indicating higher efficiency. However, the design effect of the GLS estimator is highly sensitive to model specification. If an underlying random coefficient model is incorrectly specified as a random intercept model, the design effect can be seriously understated. In contrast, the OLS estimator of the regression slope and the design effect calculated from a design-based perspective are robust to misspecification of the variance structure, making them more reliable in situations where the model specification may not be accurate.

<h2 id="uses">Uses</h2>

 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 may be used when planning a future data collection, as well as a diagnostic tool:<a class="footnote-ref" id="fnref:131" href="#fn:131">131</a>: 85 

<ul><li>When planning a future data collection - 
 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 may be used to evaluate the sampling efficiency. E.g. if there is potentially "too much" increase in variance due to some sampling design decision, or if some alternative (economically feasible) design is more efficient. This also influences the sample size (overall, per stratum, per cluster, etc.). When planning the sample size, work may be done to correct the design effect so as to separate the interviewer effect (measurement error) from the effects of the sampling design on the sampling variance.<a class="footnote-ref" id="fnref:132" href="#fn:132">132</a></li>
<li>As a diagnostic tool - 
 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 may help in evaluating potential problems with a post-hoc weighting analysis (e.g. from non-response adjustments).<a class="footnote-ref" id="fnref:133" href="#fn:133">133</a> For example, if the 
 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 value is especially high, then it might indicate an issue with the sampling or weighting scheme. This can also assist when performing some manipulation on the weights (e.g., weight trimming), the design effect could be used to evaluate the influence of the manipulation on the effective sample size.<a class="footnote-ref" id="fnref:134" href="#fn:134">134</a> And also in identifying glaring issues with the data or its analysis (e.g., ranging from mistakes to the presence of <a href="/facts/Outlier/VBhIzGbx">Outliers</a>).<a class="footnote-ref" id="fnref:135" href="#fn:135">135</a>: 191  Although some literature suggests that 
 
 
 
 D
 e
 f
 f
 >
 1.5
 
 
 {\displaystyle Deff>1.5}
 
 is likely to require some attention,<a class="footnote-ref" id="fnref:136" href="#fn:136">136</a>: 396  there is no universal rule of thumb for which design effect value is "too high". Practical considerations of 
 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 values are often context dependent.[original research?]</li></ul>
Considering the design effect is unnecessary when<a class="footnote-ref" id="fnref:137" href="#fn:137">137</a>: 57–62  the source population is closely <a href="/facts/Independent_and_identically_distributed_random_variables/othIRaWt">IID</a>, or when the sample design of the data was drawn as a <a href="/facts/Simple_random_sample/vDnCHnNx">simple random sample</a>. It is also less useful when the sample size is relatively small (at least partially, for practical reasons).[original research?]
While Kish originally hoped to have the design effect be as agnostic as possible to the underlying distribution of the data, sampling probabilities, their correlations, and the statistics of interest, followup research has shown that these do influence the design effect. Hence, these properties should be carefully considered when deciding which 
 
 
 
 D
 e
 f
 f
 
 
 {\displaystyle Deff}
 
 calculation to use, and how to use it.<a class="footnote-ref" id="fnref:138" href="#fn:138">138</a>: 13 <a class="footnote-ref" id="fnref:139" href="#fn:139">139</a>: 6 
The design effect is rarely applied when constructing confidence intervals. Ideally, one would be able to determine, for an estimator of a particular parameter, both the variance under Simple Random Sample (SRS) with replacement and the design effect (which accounts for all elements of the sampling design that change the variance). In such scenarios, the basic variance and the design effect could have been multiplied to compute the variance of the estimator for the specific design.<a class="footnote-ref" id="fnref:140" href="#fn:140">140</a>: 259  This computed value can then be employed to form confidence intervals. However, in real-world applications, it is uncommon to estimate both values simultaneously. As a result, other methods are favored. For instance, Taylor linearization is utilized to construct confidence intervals based on the <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">variance of the weighted mean</a>. More broadly, the bootstrap method, also known as <a href="/facts/Weighted_arithmetic_mean/VHyJfPxb">replication weights</a>, is applied for a range of weighted statistics.

<h2 id="software-implementations">Software implementations</h2>
Kish's design effect is implemented in various statistical software packages:

<ul><li>R: <a href="https://r-survey.r-forge.r-project.org/survey/html/surveysummary.html">surveysummary</a> from the <a href="https://cran.r-project.org/web/packages/survey/index.html">survey</a> package.<a class="footnote-ref" id="fnref:141" href="#fn:141">141</a> It is also implemented in other R packages (e.g., pewmethods,<a class="footnote-ref" id="fnref:142" href="#fn:142">142</a> and samplesize4surveys<a class="footnote-ref" id="fnref:143" href="#fn:143">143</a>).</li>
<li>Python: <a href="https://import-balance.org/api_reference/html/balance.stats_and_plots.weights_stats.html#balance.stats_and_plots.weights_stats.design_effect">design_effect</a> from the <a href="https://import-balance.org/">balance</a> package.<a class="footnote-ref" id="fnref:144" href="#fn:144">144</a></li>
<li>SAS: Using Proc Surveymeans.<a class="footnote-ref" id="fnref:145" href="#fn:145">145</a></li>
<li>Stata: Using the estat post-estimation command after the svy: mean command.<a class="footnote-ref" id="fnref:146" href="#fn:146">146</a></li></ul>
<ul><li>sudaan.<a class="footnote-ref" id="fnref:147" href="#fn:147">147</a></li>
<li>WESVAR: calculates Kish's design effect with replacement (SRSWR), i.e. 
 
 
 
 D
 e
 f
 t
 
 
 {\displaystyle Deft}
 
.<a class="footnote-ref" id="fnref:148" href="#fn:148">148</a></li></ul>
<h2 id="notes">Notes</h2>

This article was submitted to WikiJournal of Science for external <a href="/facts/Scholarly_peer_review/ufyrXAkW">academic peer review</a> in 2023 (<a href="https://en.wikiversity.org/wiki/Talk:WikiJournal_Preprints/Design_effect">reviewer reports</a>). The updated content was reintegrated into the Wikipedia page under a CC-BY-SA-3.0 license (<a href="https://en.wikipedia.org/w/index.php?title=Design_effect&action=history&date-range-to=2024-05-07">2024</a>). The version of record as reviewed is: 
Tal Galili; et al. (5 May 2024). <a href="https://upload.wikimedia.org/wikiversity/en/6/6d/Design_Effects.pdf">"Design effect"</a> (PDF). WikiJournal of Science. 7 (1): 4. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.15347%2FWJS%2F2024.004">10.15347/WJS/2024.004</a>. <a href="/facts/ISSN_(identifier)/DPAflDvU">ISSN</a> <a href="https://search.worldcat.org/issn/2470-6345">2470-6345</a>. <a href="/facts/WDQ_(identifier)/5Myy74K5">Wikidata</a> Q116768211.

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Heo, Moonseong; Kim, Yongman; Xue, Xiaonan; Kim, Mimi Y. (2010). "Sample size requirement to detect an intervention effect at the end of follow-up in a longitudinal cluster randomized trial". Statistics in Medicine. 29 (3): 382–390. doi:10.1002/sim.3806. ISSN 1097-0258. PMID 20014353. S2CID 30001378. Archived from the original on 5 January 2013. <a href="https://archive.today/20130105190734/http://www3.interscience.wiley.com/journal/123212319/abstract" target="_blank">https://archive.today/20130105190734/http://www3.interscience.wiley.com/journal/123212319/abstract</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">I.e., that the design effect is the ratio of variances of two estimators, one from a sample with some design and the other from a simple random sample <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Kish, Leslie (1995). "Methods for design effects" (PDF). Journal of Official Statistics. 11 (1): 55. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf" target="_blank">https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Cochran, William G. (June 1951). "General Principles in the Selection of a Sample". American Journal of Public Health and the Nation's Health. 41 (6): 647–653. doi:10.2105/AJPH.41.6.647. ISSN 0090-0036. PMC 1525569. PMID 14838186. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1525569" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1525569</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Kish, Leslie (1995). "Methods for design effects" (PDF). Journal of Official Statistics. 11 (1): 55. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf" target="_blank">https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Everitt, B.S. (2002). The Cambridge Dictionary of Statistics (2nd ed.). Cambridge University Press. ISBN 0-521-81099-X. <a href="0-521-81099-X" target="_blank">0-521-81099-X</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">A general formula for the (theoretical) design effect of estimating a total (not the mean), for some design, is given in Cochran 1977.[3]: 54
 <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">Kalton, Graham; Brick, J. Michael; Lȇ, Thanh (2005). Estimating components of design effects for use in sample design (PDF). Household Sample Surveys in Developing and Transition Countries (Report). New York: Department of Economic and Social Affairs, Statistics Division, United Nations. pp. 95–121. ISBN 92-1-161481-3. ST/ESA/STAT/SER.F/96. <a href="92-1-161481-3" target="_blank">92-1-161481-3</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">Kish, Leslie (1995). "Methods for design effects" (PDF). Journal of Official Statistics. 11 (1): 55. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf" target="_blank">https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">The original intention of Kish for 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 was to have it "express the effects of sample design beyond the elemental variability 
 
 
 
 
 
 
 S
 
 n
 
 
 2
 
 
 n
 
 
 
 
 {\displaystyle {\frac {S_{n}^{2}}{n}}}
 
, removing both the unit of measurement and sample size as nuisance parameters". The hope was to have the design effect generalizable (relevant for) many statistics and variables within the same survey (and even between surveys).[5]: 55  However, followup works have shown that the design effect depends on the specific sampling design, the outcome, and the statistic of interest (E.g. population total versus the mean). Especially, the 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 depends on the association between some specific outcome with a specific design (e.g. the correlation between 
 
 
 
 
 y
 
 i
 
 
 
 
 {\displaystyle y_{i}}
 
 and the selection probability 
 
 
 
 
 p
 
 i
 
 
 
 
 {\displaystyle p_{i}}
 
).[4]: 5  Hence, current literature does not support the generalizability of the 
 
 
 
 
 Deft
 
 
 
 {\displaystyle {\text{Deft}}}
 
 across many statistics and outcome measures. <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Leinster, Tom (18 December 2014). "Effective Sample Size". The n-Category Café. <a href="https://golem.ph.utexas.edu/category/2014/12/effective_sample_size.html" target="_blank">https://golem.ph.utexas.edu/category/2014/12/effective_sample_size.html</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
<li id="fn:25">Wolter, Kirk M. (2007). Introduction to Variance Estimation. Statistics for Social and Behavioral Sciences (2nd ed.). Springer. doi:10.1007/978-0-387-35099-8. ISBN 978-0387329178. <a href="978-0387329178" target="_blank">978-0387329178</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></li>
<li id="fn:26">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></li>
<li id="fn:27">As a simple illustration of this, imagine we have clusters of different sizes, and we sample only one cluster (using SRS) and measure all the elements in it. This will lead to EPSEM, but the number of observations we'll get will depend on the cluster size. <a href="#fnref:27" class="footnote-back-ref">↩</a></li>
<li id="fn:28">Frerichs, R. R. (2004). "Equal Probability of Selection". Rapid Surveys (PDF). unpublished. <a href="https://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chap4rapid_2004.pdf" target="_blank">https://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chap4rapid_2004.pdf</a> <a href="#fnref:28" class="footnote-back-ref">↩</a></li>
<li id="fn:29">To be more precise: suppose that 
 
 
 
 
 S
 
 i
 
 
 
 
 {\displaystyle S_{i}}
 
 is the measure of size for cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
. One common method of PPS (probability proportional to size) sampling is to sample each cluster with selection probability that is proportional to its size as follows: 
 
 
 
 P
 (
 
 Selecting cluster 
 
 i
 )
 =
 
 
 
 m
 
 S
 
 i
 
 
 
 
 
 ∑
 
 
 U
 
 i
 
 
 ∈
 U
 
 
 
 S
 
 i
 
 
 
 
 
 
 
 {\displaystyle P({\text{Selecting cluster }}i)={\frac {mS_{i}}{\sum _{U_{i}\in U}S_{i}}}}
 
 where 
 
 
 
 m
 
 
 {\displaystyle m}
 
 is the number of clusters that we want to sample and 
 
 
 
 U
 
 
 {\displaystyle U}
 
 is the frame used for sampling clusters. If we subsampled an equal number, 
 
 
 
 
 
 
 n
 ¯
 
 
 
 
 
 {\displaystyle {\bar {n}}}
 
, of elements within each sample cluster using some equal probability method, and 
 
 
 
 
 S
 
 i
 
 
 
 
 {\displaystyle S_{i}}
 
 is the correct number of elements in cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
, then the selection probability of element 
 
 
 
 j
 
 
 {\displaystyle j}
 
 (in some cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
) will be the same for every element across all clusters (i.e., EPSEM):

π
          
            j
          
        
        =
        
          
            
              m
              
                S
                
                  i
                
              
            
            
              
                ∑
                
                  
                    U
                    
                      i
                    
                  
                  ∈
                  U
                
              
              
                S
                
                  i
                
              
            
          
        
        
          
            
              
                n
                ¯
              
            
            
              S
              
                i
              
            
          
        
        =
        
          
            
              m
              
                
                  
                    n
                    ¯
                  
                
              
            
            
              
                ∑
                
                  
                    U
                    
                      i
                    
                  
                  ∈
                  U
                
              
              
                S
                
                  i
                
              
            
          
        
      
    
    {\displaystyle \pi _{j}={\frac {mS_{i}}{\sum _{U_{i}\in U}S_{i}}}{\frac {\bar {n}}{S_{i}}}={\frac {m{\bar {n}}}{\sum _{U_{i}\in U}S_{i}}}}
  
.

If 
 
 
 
 
 S
 
 i
 
 
 
 
 {\displaystyle S_{i}}
 
 turns out not to be the correct size, sampling at the rate of 
 
 
 
 
 
 
 
 n
 ¯
 
 
 
 S
 
 i
 
 
 
 
 
 
 {\displaystyle {\frac {\bar {n}}{S_{i}}}}
 
 will still yield EPSEM (equal probability selection method). Notice that if we enumerate (take measurement of) all units in a sample cluster (instead of some fixed number 
 
 
 
 
 
 
 n
 ¯
 
 
 
 
 
 {\displaystyle {\bar {n}}}
 
, or a fixed proportion 
 
 
 
 
 
 
 
 n
 ¯
 
 
 
 S
 
 i
 
 
 
 
 
 
 {\displaystyle {\frac {\bar {n}}{S_{i}}}}
 
), then each unit in cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
 has the selection probability of the cluster, which will lead to unequal probability of selections between elements of different clusters (i.e., 
 
 
 
 
 π
 
 j
 
 
 (
 i
 )
 =
 
 
 
 m
 
 S
 
 i
 
 
 
 
 
 ∑
 
 
 U
 
 i
 
 
 ∈
 U
 
 
 
 S
 
 i
 
 
 
 
 
 
 
 {\displaystyle \pi _{j}(i)={\frac {mS_{i}}{\sum _{U_{i}\in U}S_{i}}}}
 
). <a href="#fnref:29" class="footnote-back-ref">↩</a></li>
<li id="fn:30">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:30" class="footnote-back-ref">↩</a></li>
<li id="fn:31">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:31" class="footnote-back-ref">↩</a></li>
<li id="fn:32">Kish, Leslie (1995). "Methods for design effects" (PDF). Journal of Official Statistics. 11 (1): 55. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf" target="_blank">https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf</a> <a href="#fnref:32" class="footnote-back-ref">↩</a></li>
<li id="fn:33">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:33" class="footnote-back-ref">↩</a></li>
<li id="fn:34">Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Nashville, TN: John Wiley & Sons. ISBN 978-0-471-16240-7. <a href="978-0-471-16240-7" target="_blank">978-0-471-16240-7</a> <a href="#fnref:34" class="footnote-back-ref">↩</a></li>
<li id="fn:35">Neyman, Jerzy (1934). "On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection". Journal of the Royal Statistical Society. 97 (4): 558–625. doi:10.2307/2342192. ISSN 0952-8385. JSTOR 2342192. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:35" class="footnote-back-ref">↩</a></li>
<li id="fn:36">Kalton, Graham; Brick, J. Michael; Lȇ, Thanh (2005). Estimating components of design effects for use in sample design (PDF). Household Sample Surveys in Developing and Transition Countries (Report). New York: Department of Economic and Social Affairs, Statistics Division, United Nations. pp. 95–121. ISBN 92-1-161481-3. ST/ESA/STAT/SER.F/96. <a href="92-1-161481-3" target="_blank">92-1-161481-3</a> <a href="#fnref:36" class="footnote-back-ref">↩</a></li>
<li id="fn:37">For example, say that we assume for each cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
 that its size is 
 
 
 
 
 S
 
 i
 
 
 
 
 {\displaystyle S_{i}}
 
, we can sample 
 
 
 
 m
 
 
 {\displaystyle m}
 
 clusters with the following probability of selection: 
 
 
 
 P
 (
 
 Selecting cluster 
 
 i
 )
 =
 
 
 
 m
 
 S
 
 i
 
 
 
 
 
 ∑
 
 
 U
 
 i
 
 
 ∈
 U
 
 
 
 S
 
 i
 
 
 
 
 
 
 
 {\displaystyle P({\text{Selecting cluster }}i)={\frac {mS_{i}}{\sum _{U_{i}\in U}S_{i}}}}
 
. And then, we take a fixed number of 
 
 
 
 
 
 
 n
 ¯
 
 
 
 
 
 {\displaystyle {\bar {n}}}
 
 elements from each cluster. In such a case, if we say that the real cluster size is, say, 
 
 
 
 
 S
 
 i
 
 
 ∗
 
 
 
 
 {\displaystyle S_{i}^{*}}
 
, then the selection probability for each element 
 
 
 
 j
 
 
 {\displaystyle j}
 
 taken from cluster 
 
 
 
 i
 
 
 {\displaystyle i}
 
, will be: 
 
 
 
 
 π
 
 j
 
 
 (
 j
 )
 =
 
 
 
 m
 
 S
 
 i
 
 
 
 
 
 ∑
 
 
 U
 
 i
 
 
 ∈
 U
 
 
 
 S
 
 i
 
 
 
 
 
 
 
 
 
 n
 ¯
 
 
 
 S
 
 i
 
 
 ∗
 
 
 
 
 
 
 {\displaystyle \pi _{j}(j)={\frac {mS_{i}}{\sum _{U_{i}\in U}S_{i}}}{\frac {\bar {n}}{S_{i}^{*}}}}
 
. Note that this could be mitigated at the sampling stage if we sample from each cluster using the rate 
 
 
 
 
 
 
 
 n
 ¯
 
 
 
 S
 
 i
 
 
 
 
 
 
 {\displaystyle {\frac {\bar {n}}{S_{i}}}}
 
, then the selection probability will be EPSEM (even though the real cluster size was 
 
 
 
 
 S
 
 i
 
 
 ∗
 
 
 
 
 {\displaystyle S_{i}^{*}}
 
 and not 
 
 
 
 
 S
 
 i
 
 
 
 
 {\displaystyle S_{i}}
 
). <a href="#fnref:37" class="footnote-back-ref">↩</a></li>
<li id="fn:38">Frerichs, R. R. (2004). "Equal Probability of Selection". Rapid Surveys (PDF). unpublished. <a href="https://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chap4rapid_2004.pdf" target="_blank">https://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chap4rapid_2004.pdf</a> <a href="#fnref:38" class="footnote-back-ref">↩</a></li>
<li id="fn:39">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:39" class="footnote-back-ref">↩</a></li>
<li id="fn:40">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:40" class="footnote-back-ref">↩</a></li>
<li id="fn:41">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:41" class="footnote-back-ref">↩</a></li>
<li id="fn:42">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:42" class="footnote-back-ref">↩</a></li>
<li id="fn:43">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:43" class="footnote-back-ref">↩</a></li>
<li id="fn:44">Dever, Jill A.; Valliant, Richard (2010). "A comparison of variance estimators for post-stratification to estimated control totals" (PDF). Survey Methodology. 36 (1): 45–56. ISSN 1492-0921. <a href="https://www.rti.org/publication/comparison-variance-estimators-poststratification-estimated-control-totals/fulltext.pdf" target="_blank">https://www.rti.org/publication/comparison-variance-estimators-poststratification-estimated-control-totals/fulltext.pdf</a> <a href="#fnref:44" class="footnote-back-ref">↩</a></li>
<li id="fn:45">Kott, Phillip S. (2006). "Using calibration weighting to adjust for nonresponse and coverage errors" (PDF). Survey Methodology. 32 (2): 133. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf</a> <a href="#fnref:45" class="footnote-back-ref">↩</a></li>
<li id="fn:46">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:46" class="footnote-back-ref">↩</a></li>
<li id="fn:47">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:47" class="footnote-back-ref">↩</a></li>
<li id="fn:48">Holt, D.; Smith, T. M. F. (1979). "Post Stratification". Journal of the Royal Statistical Society. Series A (General). 142 (1): 33–46. doi:10.2307/2344652. ISSN 0035-9238. JSTOR 2344652. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:48" class="footnote-back-ref">↩</a></li>
<li id="fn:49">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:49" class="footnote-back-ref">↩</a></li>
<li id="fn:50">This formula would apply only if an equal probability sample were selected in stratum h and each element has the same probability of responding. <a href="#fnref:50" class="footnote-back-ref">↩</a></li>
<li id="fn:51">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:51" class="footnote-back-ref">↩</a></li>
<li id="fn:52">Ghosh, Dhiren; Vogt, Andrew (2002). "Sampling methods related to Bernoulli and Poisson Sampling" (PDF). Proceedings of the Section on Survey Research Methods. 2002: 3569–3570. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001080.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001080.pdf</a> <a href="#fnref:52" class="footnote-back-ref">↩</a></li>
<li id="fn:53">Deville, Jean-Claude; Särndal, Carl-Erik (1992). "Calibration Estimators in Survey Sampling". Journal of the American Statistical Association. 87 (418): 376–382. doi:10.1080/01621459.1992.10475217. ISSN 0162-1459. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:53" class="footnote-back-ref">↩</a></li>
<li id="fn:54">Kott, Phillip S. (2006). "Using calibration weighting to adjust for nonresponse and coverage errors" (PDF). Survey Methodology. 32 (2): 133. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf</a> <a href="#fnref:54" class="footnote-back-ref">↩</a></li>
<li id="fn:55">Brick, J. Michael; Montaquila, Jill; Roth, Shelley (2003). "Identifying problems with raking estimators" (PDF). Proceedings of the Section on Survey Research Methods. 2003: 710–717. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2003/Files/JSM2003-000472.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2003/Files/JSM2003-000472.pdf</a> <a href="#fnref:55" class="footnote-back-ref">↩</a></li>
<li id="fn:56">Kott, Phillip S. (2006). "Using calibration weighting to adjust for nonresponse and coverage errors" (PDF). Survey Methodology. 32 (2): 133. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf</a> <a href="#fnref:56" class="footnote-back-ref">↩</a></li>
<li id="fn:57">Keiding, Niels; Clayton, David (2014). "Standardization and Control for Confounding in Observational Studies: A Historical Perspective". Statistical Science. 29 (4): 529–558. arXiv:1503.02853. doi:10.1214/13-STS453. ISSN 0883-4237. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:57" class="footnote-back-ref">↩</a></li>
<li id="fn:58">Lumley, Thomas (25 May 2021). "How to estimate the (approximate) variance of the weighted mean?". Stack Exchange. <a href="https://stats.stackexchange.com/q/525770" target="_blank">https://stats.stackexchange.com/q/525770</a> <a href="#fnref:58" class="footnote-back-ref">↩</a></li>
<li id="fn:59">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:59" class="footnote-back-ref">↩</a></li>
<li id="fn:60">"What types of weights do SAS, Stata and SPSS support?". UCLA Statistical Consulting Group. 2021. Archived from the original on 2 September 2023. Retrieved 2 September 2023. <a href="https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/" target="_blank">https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/</a> <a href="#fnref:60" class="footnote-back-ref">↩</a></li>
<li id="fn:61">"What types of weights do SAS, Stata and SPSS support?". UCLA Statistical Consulting Group. 2021. Archived from the original on 2 September 2023. Retrieved 2 September 2023. <a href="https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/" target="_blank">https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/</a> <a href="#fnref:61" class="footnote-back-ref">↩</a></li>
<li id="fn:62">Kalton, Graham (1968). "Standardization: A Technique to Control for Extraneous Variables". Journal of the Royal Statistical Society. Series C (Applied Statistics). 17 (2): 118–136. doi:10.2307/2985676. ISSN 0035-9254. JSTOR 2985676. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:62" class="footnote-back-ref">↩</a></li>
<li id="fn:63">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:63" class="footnote-back-ref">↩</a></li>
<li id="fn:64">"What types of weights do SAS, Stata and SPSS support?". UCLA Statistical Consulting Group. 2021. Archived from the original on 2 September 2023. Retrieved 2 September 2023. <a href="https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/" target="_blank">https://web.archive.org/web/20230902175545/https://stats.oarc.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/</a> <a href="#fnref:64" class="footnote-back-ref">↩</a></li>
<li id="fn:65">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:65" class="footnote-back-ref">↩</a></li>
<li id="fn:66">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:66" class="footnote-back-ref">↩</a></li>
<li id="fn:67">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:67" class="footnote-back-ref">↩</a></li>
<li id="fn:68">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:68" class="footnote-back-ref">↩</a></li>
<li id="fn:69">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:69" class="footnote-back-ref">↩</a></li>
<li id="fn:70">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:70" class="footnote-back-ref">↩</a></li>
<li id="fn:71">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:71" class="footnote-back-ref">↩</a></li>
<li id="fn:72">Henry, Kimberly A.; Valliant, Richard (2015). "A design effect measure for calibration weighting in single-stage samples" (PDF). Survey Methodology. 41 (2): 315–331. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf</a> <a href="#fnref:72" class="footnote-back-ref">↩</a></li>
<li id="fn:73">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:73" class="footnote-back-ref">↩</a></li>
<li id="fn:74">Bock, Tim (24 March 2017). "Design Effects and Effective Sample Size". Displayr. <a href="http://docs.displayr.com/wiki/Design_Effects_and_Effective_Sample_Size" target="_blank">http://docs.displayr.com/wiki/Design_Effects_and_Effective_Sample_Size</a> <a href="#fnref:74" class="footnote-back-ref">↩</a></li>
<li id="fn:75">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:75" class="footnote-back-ref">↩</a></li>
<li id="fn:76">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:76" class="footnote-back-ref">↩</a></li>
<li id="fn:77">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:77" class="footnote-back-ref">↩</a></li>
<li id="fn:78">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:78" class="footnote-back-ref">↩</a></li>
<li id="fn:79">Little, Roderick J.; Vartivarian, Sonya (2005). "Does weighting for nonresponse increase the variance of survey means?" (PDF). Survey Methodology. 31 (2): 161. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2005002/article/9046-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2005002/article/9046-eng.pdf</a> <a href="#fnref:79" class="footnote-back-ref">↩</a></li>
<li id="fn:80">Notice that there is another term called relative variance, which is different. It is the ratio of variance to the mean, while Kish's relvariance is the ratio of the variance to the squared mean. <a href="/wiki/Index_of_dispersion" target="_blank">/wiki/Index_of_dispersion</a> <a href="#fnref:80" class="footnote-back-ref">↩</a></li>
<li id="fn:81">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:81" class="footnote-back-ref">↩</a></li>
<li id="fn:82">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:82" class="footnote-back-ref">↩</a></li>
<li id="fn:83">Henry, Kimberly A.; Valliant, Richard (2015). "A design effect measure for calibration weighting in single-stage samples" (PDF). Survey Methodology. 41 (2): 315–331. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf</a> <a href="#fnref:83" class="footnote-back-ref">↩</a></li>
<li id="fn:84">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:84" class="footnote-back-ref">↩</a></li>
<li id="fn:85">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:85" class="footnote-back-ref">↩</a></li>
<li id="fn:86">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:86" class="footnote-back-ref">↩</a></li>
<li id="fn:87">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:87" class="footnote-back-ref">↩</a></li>
<li id="fn:88">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:88" class="footnote-back-ref">↩</a></li>
<li id="fn:89">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:89" class="footnote-back-ref">↩</a></li>
<li id="fn:90">Henry, Kimberly A.; Valliant, Richard (2015). "A design effect measure for calibration weighting in single-stage samples" (PDF). Survey Methodology. 41 (2): 315–331. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf</a> <a href="#fnref:90" class="footnote-back-ref">↩</a></li>
<li id="fn:91">Liu, Jun; Iannacchione, Vince; Byron, Margie (2002). "Decomposing design effects for stratified sampling" (PDF). Proceedings of the Section on Survey Research Methods. 2002: 2124–2126. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf</a> <a href="#fnref:91" class="footnote-back-ref">↩</a></li>
<li id="fn:92">In the literature, the sample and the population sizes are sometimes marked as n and N, and sometimes m and M. In this article we used n and N. <a href="#fnref:92" class="footnote-back-ref">↩</a></li>
<li id="fn:93">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:93" class="footnote-back-ref">↩</a></li>
<li id="fn:94">Spencer, Bruce D. (2000). "An approximate design effect for unequal weighting when measurements may correlate with selection probabilities" (PDF). Survey Methodology. 26: 137–138. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p</a> <a href="#fnref:94" class="footnote-back-ref">↩</a></li>
<li id="fn:95">Spencer, Bruce D. (2000). "An approximate design effect for unequal weighting when measurements may correlate with selection probabilities" (PDF). Survey Methodology. 26: 137–138. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p</a> <a href="#fnref:95" class="footnote-back-ref">↩</a></li>
<li id="fn:96">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:96" class="footnote-back-ref">↩</a></li>
<li id="fn:97">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:97" class="footnote-back-ref">↩</a></li>
<li id="fn:98">Spencer, Bruce D. (2000). "An approximate design effect for unequal weighting when measurements may correlate with selection probabilities" (PDF). Survey Methodology. 26: 137–138. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=t-Ccnb4p</a> <a href="#fnref:98" class="footnote-back-ref">↩</a></li>
<li id="fn:99">Henry, Kimberly A.; Valliant, Richard (2015). "A design effect measure for calibration weighting in single-stage samples" (PDF). Survey Methodology. 41 (2): 315–331. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf</a> <a href="#fnref:99" class="footnote-back-ref">↩</a></li>
<li id="fn:100">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:100" class="footnote-back-ref">↩</a></li>
<li id="fn:101">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:101" class="footnote-back-ref">↩</a></li>
<li id="fn:102">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:102" class="footnote-back-ref">↩</a></li>
<li id="fn:103">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:103" class="footnote-back-ref">↩</a></li>
<li id="fn:104">Rowe, Alexander K.; Lama, Marcel; Onikpo, Faustin; Deming, Michael S. (2002). "Design effects and intraclass correlation coefficients from a health facility cluster survey in Benin". International Journal for Quality in Health Care. 14 (6): 521–523. doi:10.1093/intqhc/14.6.521. ISSN 1353-4505. PMID 12515339. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:104" class="footnote-back-ref">↩</a></li>
<li id="fn:105">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:105" class="footnote-back-ref">↩</a></li>
<li id="fn:106">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:106" class="footnote-back-ref">↩</a></li>
<li id="fn:107">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:107" class="footnote-back-ref">↩</a></li>
<li id="fn:108">Bland, Michael (2005). "Cluster randomised trials in the medical literature". University of York. <a href="http://www-users.york.ac.uk/~mb55/talks/clusml.htm" target="_blank">http://www-users.york.ac.uk/~mb55/talks/clusml.htm</a> <a href="#fnref:108" class="footnote-back-ref">↩</a></li>
<li id="fn:109">Ahmed, Saifuddin (2009). "Methods in Sample Surveys" (PDF). Johns Hopkins University Bloomberg School of Public Health. pp. 5–6. Archived from the original (PDF) on 28 September 2013. <a href="https://web.archive.org/web/20130928180152/https://ocw.jhsph.edu/courses/StatMethodsForSampleSurveys/PDFs/Lecture5.pdf" target="_blank">https://web.archive.org/web/20130928180152/https://ocw.jhsph.edu/courses/StatMethodsForSampleSurveys/PDFs/Lecture5.pdf</a> <a href="#fnref:109" class="footnote-back-ref">↩</a></li>
<li id="fn:110">Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Nashville, TN: John Wiley & Sons. ISBN 978-0-471-16240-7. <a href="978-0-471-16240-7" target="_blank">978-0-471-16240-7</a> <a href="#fnref:110" class="footnote-back-ref">↩</a></li>
<li id="fn:111">Liu, Jun; Iannacchione, Vince; Byron, Margie (2002). "Decomposing design effects for stratified sampling" (PDF). Proceedings of the Section on Survey Research Methods. 2002: 2124–2126. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf</a> <a href="#fnref:111" class="footnote-back-ref">↩</a></li>
<li id="fn:112">Kish, Leslie (1987). "Questions/Answers" (PDF). The Survey Statistician. Vol. 17. pp. 13–17. ISSN 0214-3240. <a href="https://commons.wikimedia.org/wiki/File:Leslie_Kish,_1987,_Survey_Statistician_volume_17.pdf" target="_blank">https://commons.wikimedia.org/wiki/File:Leslie_Kish,_1987,_Survey_Statistician_volume_17.pdf</a> <a href="#fnref:112" class="footnote-back-ref">↩</a></li>
<li id="fn:113">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:113" class="footnote-back-ref">↩</a></li>
<li id="fn:114">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:114" class="footnote-back-ref">↩</a></li>
<li id="fn:115">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:115" class="footnote-back-ref">↩</a></li>
<li id="fn:116">Lynn, Peter; Gabler, Siegfried (2005). "Approximations to b* in the prediction of design effects due to clustering" (PDF). Survey Methodology. 31 (1): 101–104. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2005001/article/8093-eng.pdf?st=J-njxreT" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2005001/article/8093-eng.pdf?st=J-njxreT</a> <a href="#fnref:116" class="footnote-back-ref">↩</a></li>
<li id="fn:117">Kish, Leslie (1987). "Questions/Answers" (PDF). The Survey Statistician. Vol. 17. pp. 13–17. ISSN 0214-3240. <a href="https://commons.wikimedia.org/wiki/File:Leslie_Kish,_1987,_Survey_Statistician_volume_17.pdf" target="_blank">https://commons.wikimedia.org/wiki/File:Leslie_Kish,_1987,_Survey_Statistician_volume_17.pdf</a> <a href="#fnref:117" class="footnote-back-ref">↩</a></li>
<li id="fn:118">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:118" class="footnote-back-ref">↩</a></li>
<li id="fn:119">Gabler, Siegfried; Hader, Sabine; Lynn, Peter (2005). "Design effects for multiple design samples" (PDF). Survey Methodology. 32 (1): 115–120. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006001/article/9256-eng.pdf?st=YXTS--Q-" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006001/article/9256-eng.pdf?st=YXTS--Q-</a> <a href="#fnref:119" class="footnote-back-ref">↩</a></li>
<li id="fn:120">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:120" class="footnote-back-ref">↩</a></li>
<li id="fn:121">The formula for Kish's design effect using the original notation:[36]: 16

Deff
 
 
 Kish
 
 
 =
 
 
 Deft
 
 
 2
 
 
 =
 d
 e
 f
 t
 
 u
 
 s
 
 
 2
 
 
 (
 1
 +
 L
 )
 =
 
 (
 
 1
 +
 ρ
 (
 
 
 
 b
 ¯
 
 
 
 −
 1
 )
 
 )
 
 
 
 
 n
 ∑
 
 
 k
 
 j
 
 
 2
 
 
 
 
 
 
 (
 
 ∑
 
 
 k
 
 j
 
 
 
 
 )
 
 
 2
 
 
 
 
 
 
 {\displaystyle {\text{Deff}}_{\text{Kish}}={\text{Deft}}^{2}=deftu_{s}^{2}(1+L)=\left(1+\rho ({\bar {b}}-1)\right){\frac {n\sum {k_{j}^{2}}}{\left(\sum {k_{j}}\right)^{2}}}}
 <a href="#fnref:121" class="footnote-back-ref">↩</a></li>
<li id="fn:122">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:122" class="footnote-back-ref">↩</a></li>
<li id="fn:123">Liu, Jun; Aragon, Elvessa (2000). "Subsampling strategies in longitudinal surveys" (PDF). Proceedings of the Section on Survey Research Methods. 2000: 307–312. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/papers/2000_048.pdf" target="_blank">http://www.asasrms.org/Proceedings/papers/2000_048.pdf</a> <a href="#fnref:123" class="footnote-back-ref">↩</a></li>
<li id="fn:124">Liu, Jun; Iannacchione, Vince; Byron, Margie (2002). "Decomposing design effects for stratified sampling" (PDF). Proceedings of the Section on Survey Research Methods. 2002: 2124–2126. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2002/Files/JSM2002-001069.pdf</a> <a href="#fnref:124" class="footnote-back-ref">↩</a></li>
<li id="fn:125">Park, Inho; Winglee, Marianne; Clark, Jay; Rust, Keith; Sedlak, Andrea; Morganstein, David (2003). "Design effects and survey planning" (PDF). Proceedings of the Section on Survey Research Methods. 2003: 3179–3186. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2003/Files/JSM2003-000820.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2003/Files/JSM2003-000820.pdf</a> <a href="#fnref:125" class="footnote-back-ref">↩</a></li>
<li id="fn:126">Gabler, Siegfried; Häder, Sabine; Lahiri, Partha (1999). "A model based justification of Kish's formula for design effects for weighting and clustering" (PDF). Survey Methodology. 25: 105–106. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1999001/article/4718-eng.pdf?st=kP7KrrRP</a> <a href="#fnref:126" class="footnote-back-ref">↩</a></li>
<li id="fn:127">Chen, Sixia; Rust, Keith (2017). "An extension of Kish's formula for design effects to two-and three-stage designs with stratification". Journal of Survey Statistics and Methodology. 5 (2): 111–130. doi:10.1093/jssam/smw036. ISSN 2325-0984. PMC 10426793. PMID 37583392. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10426793" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10426793</a> <a href="#fnref:127" class="footnote-back-ref">↩</a></li>
<li id="fn:128">Henry, Kimberly A.; Valliant, Richard (2015). "A design effect measure for calibration weighting in single-stage samples" (PDF). Survey Methodology. 41 (2): 315–331. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2015002/article/14236-eng.pdf</a> <a href="#fnref:128" class="footnote-back-ref">↩</a></li>
<li id="fn:129">Sarndal, Carl-Erik; Swensson, Bengt; Wretman, Jan (1992). Model Assisted Survey Sampling. Springer. doi:10.1007/978-1-4612-4378-6 (inactive 1 November 2024). ISBN 9780387975283.{{cite book}}: CS1 maint: DOI inactive as of November 2024 (link) <a href="9780387975283" target="_blank">9780387975283</a> <a href="#fnref:129" class="footnote-back-ref">↩</a></li>
<li id="fn:130">Lohr, Sharon L. (2014). "Design Effects for a Regression Slope in a Cluster Sample". Journal of Survey Statistics and Methodology. 2 (2): 97–125. doi:10.1093/jssam/smu003. ISSN 2325-0984. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:130" class="footnote-back-ref">↩</a></li>
<li id="fn:131">Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Nashville, TN: John Wiley & Sons. ISBN 978-0-471-16240-7. <a href="978-0-471-16240-7" target="_blank">978-0-471-16240-7</a> <a href="#fnref:131" class="footnote-back-ref">↩</a></li>
<li id="fn:132">Zins, Stefan; Burgard, Jan Pablo (2020). "Considering interviewer and design effects when planning sample sizes". Survey Methodology. 46 (1): 93–119. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/pub/12-001-x/2020001/article/00005-eng.htm" target="_blank">https://www150.statcan.gc.ca/n1/pub/12-001-x/2020001/article/00005-eng.htm</a> <a href="#fnref:132" class="footnote-back-ref">↩</a></li>
<li id="fn:133">Kalton, Graham; Brick, J. Michael; Lȇ, Thanh (2005). Estimating components of design effects for use in sample design (PDF). Household Sample Surveys in Developing and Transition Countries (Report). New York: Department of Economic and Social Affairs, Statistics Division, United Nations. pp. 95–121. ISBN 92-1-161481-3. ST/ESA/STAT/SER.F/96. <a href="92-1-161481-3" target="_blank">92-1-161481-3</a> <a href="#fnref:133" class="footnote-back-ref">↩</a></li>
<li id="fn:134">Potter, Frank; Zheng, Yuhong (2015). "Methods and issues in trimming extreme weights in sample surveys" (PDF). Proceedings of the Section on Survey Research Methods. 2015: 2707–2719. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2015/files/234115.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2015/files/234115.pdf</a> <a href="#fnref:134" class="footnote-back-ref">↩</a></li>
<li id="fn:135">Kish, Leslie (1992). "Weighting for unequal Pi" (PDF). Journal of Official Statistics. 8 (2): 183–200. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf" target="_blank">https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/weighting-for-unequal-empemsubemiemsub.pdf</a> <a href="#fnref:135" class="footnote-back-ref">↩</a></li>
<li id="fn:136">Valliant, Richard; Dever, Jill A.; Kreuter, Frauke (2013). Practical Tools for Designing and Weighting Survey Samples. New York: Springer. doi:10.1007/978-1-4614-6449-5. ISBN 978-1-4899-9381-6. <a href="978-1-4899-9381-6" target="_blank">978-1-4899-9381-6</a> <a href="#fnref:136" class="footnote-back-ref">↩</a></li>
<li id="fn:137">Kish, Leslie (1995). "Methods for design effects" (PDF). Journal of Official Statistics. 11 (1): 55. ISSN 0282-423X. <a href="https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf" target="_blank">https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/methods-for-design-effects.pdf</a> <a href="#fnref:137" class="footnote-back-ref">↩</a></li>
<li id="fn:138">Park, Inho; Lee, Hyunshik (2004). "Design effects for the weighted mean and total estimators under complex survey sampling" (PDF). Survey Methodology. 30 (2): 183–193. ISSN 1492-0921. <a href="https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2004002/article/7751-eng.pdf?st=DUPH-397</a> <a href="#fnref:138" class="footnote-back-ref">↩</a></li>
<li id="fn:139">Park, Inho; Lee, Hyunshik (2001). "The design effect: do we know all about it" (PDF). Proceedings of the Section on Survey Research Methods. 2001. ISSN 0733-5830. <a href="http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf" target="_blank">http://www.asasrms.org/Proceedings/y2001/Proceed/00144.pdf</a> <a href="#fnref:139" class="footnote-back-ref">↩</a></li>
<li id="fn:140">Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN 0-471-10949-5. <a href="0-471-10949-5" target="_blank">0-471-10949-5</a> <a href="#fnref:140" class="footnote-back-ref">↩</a></li>
<li id="fn:141">Lumley, Thomas (2004). "Analysis of Complex Survey Samples". Journal of Statistical Software. 9 (1): 1–19. doi:10.18637/jss.v009.i08. ISSN 1548-7660. R package version 2.2 <a href="https://doi.org/10.18637%2Fjss.v009.i08" target="_blank">https://doi.org/10.18637%2Fjss.v009.i08</a> <a href="#fnref:141" class="footnote-back-ref">↩</a></li>
<li id="fn:142">Pew Research Center. "pewmethods". GitHub. Retrieved 28 November 2023. <a href="https://github.com/pewresearch/pewmethods" target="_blank">https://github.com/pewresearch/pewmethods</a> <a href="#fnref:142" class="footnote-back-ref">↩</a></li>
<li id="fn:143">Gutierrez Rojas, Hugo Andres (17 January 2020). "samplesize4surveys". The Comprehensive R Archive Network (CRAN). Retrieved 28 November 2023. <a href="https://cran.r-project.org/web/packages/samplesize4surveys/index.html" target="_blank">https://cran.r-project.org/web/packages/samplesize4surveys/index.html</a> <a href="#fnref:143" class="footnote-back-ref">↩</a></li>
<li id="fn:144">Sarig, Tal; Galili, Tal; Eilat, Roee (2023). "balance -- a Python package for balancing biased data samples". arXiv:2307.06024 [stat.CO]. <a href="/wiki/ArXiv_(identifier)" target="_blank">/wiki/ArXiv_(identifier)</a> <a href="#fnref:144" class="footnote-back-ref">↩</a></li>
<li id="fn:145">Buskirk, Trent D. (2011). Estimating Design Effects for Means, Proportions and Totals from Complex Sample Survey Data Using SAS® Proc Surveymeans (PDF). Midwest SAS Users Group Conference 2011. Saint Louis, MO: Saint Louis University School of Public Health. pp. 1–13. Archived from the original (PDF) on 11 May 2015. Retrieved 28 November 2023. <a href="https://web.archive.org/web/20150511160802/https://www.mwsug.org/proceedings/2011/stats/MWSUG-2011-SA11.pdf" target="_blank">https://web.archive.org/web/20150511160802/https://www.mwsug.org/proceedings/2011/stats/MWSUG-2011-SA11.pdf</a> <a href="#fnref:145" class="footnote-back-ref">↩</a></li>
<li id="fn:146">"Survey Data Analysis in Stata 17". UCLA Statistical Consulting Group. 2021. Archived from the original on 7 June 2023. Retrieved 28 November 2023. <a href="https://web.archive.org/web/20230607050827/https://stats.oarc.ucla.edu/stata/seminars/survey-data-analysis-in-stata-17/" target="_blank">https://web.archive.org/web/20230607050827/https://stats.oarc.ucla.edu/stata/seminars/survey-data-analysis-in-stata-17/</a> <a href="#fnref:146" class="footnote-back-ref">↩</a></li>
<li id="fn:147">"DESCRIPT Example 1" (PDF). RTI International. Retrieved 28 November 2023. <a href="https://sudaanorder.rti.org/examples/DESCRIPT%20Example%201.pdf" target="_blank">https://sudaanorder.rti.org/examples/DESCRIPT%20Example%201.pdf</a> <a href="#fnref:147" class="footnote-back-ref">↩</a></li>
<li id="fn:148">Choudhry, G. Hussain; Valliant, Richard (2002). WesVar: Software for complex survey data analysis (PDF). Statistics Canada Symposium. Ottawa: Statistics Canada. Retrieved 28 November 2023. <a href="https://www150.statcan.gc.ca/n1/en/pub/11-522-x/2002001/session6/6728-eng.pdf?st=do0zuOhp" target="_blank">https://www150.statcan.gc.ca/n1/en/pub/11-522-x/2002001/session6/6728-eng.pdf?st=do0zuOhp</a> <a href="#fnref:148" class="footnote-back-ref">↩</a></li>
</ol>

Design effect open-in-new

Design effect