Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Randomized experiment
open-in-new
<h2 id="overview">Overview</h2> <p>In the statistical theory of <a href="/facts/Design_of_experiments/VMrNftbr">design of experiments</a>, randomization involves randomly allocating the experimental units across the <a href="/facts/Treatment_groups/zeT4uJ2S">treatment groups</a>. For example, if an experiment compares a new drug against a standard drug, then the patients should be allocated to either the new drug or to the standard drug control using randomization. </p><p>Randomized experimentation is <i>not</i> haphazard. Randomization reduces <a href="/facts/Bias/geuh5KrE">bias</a> by equalising other factors that have not been explicitly accounted for in the experimental design (according to the <a href="/facts/Law_of_large_numbers/X3Bjcy3v">law of large numbers</a>). Randomization also produces <a href="/facts/Ignorability/0Xgmh4Th">ignorable designs</a>, which are valuable in <a href="/facts/Statistical_model/jKkT5ftm">model</a>-based <a href="/facts/Statistical_inference/sJLjubxm">statistical inference</a>, especially <a href="/facts/Bayesian_inference/tEpK3zLx">Bayesian</a> or <a href="/facts/Likelihood_function/1L8P4NBX">likelihood</a>-based. In the design of experiments, the simplest design for comparing treatments is the "completely randomized design". Some "restriction on randomization" can occur with <a href="/facts/Blocking_(statistics)/EAwgBo1y">blocking</a> and experiments that have hard-to-change factors; additional restrictions on randomization can occur when a full randomization is infeasible or when it is desirable to reduce the <a href="/facts/Variance/ULBJKXD1">variance</a> of estimators of selected effects. </p><p>Randomization of treatment in <a href="/facts/Clinical_trials/q8LMr832">clinical trials</a> pose ethical problems. In some cases, randomization reduces the therapeutic options for both physician and patient, and so randomization requires <a href="/facts/Clinical_equipoise/f2G6LPAd">clinical equipoise</a> regarding the treatments. </p> <h2 id="online-randomized-controlled-experiments">Online randomized controlled experiments</h2> <p>Web sites can run randomized controlled experiments<a class="footnote-ref" id="fnref:1" href="#fn:1"><sup>1</sup></a> to create a feedback loop.<a class="footnote-ref" id="fnref:2" href="#fn:2"><sup>2</sup></a> Key differences between offline experimentation and online experiments include:<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a><a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> </p> <ul><li>Logging: user interactions can be logged reliably.</li> <li>Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.</li> <li>Number of concurrent experiments: large sites run tens of overlapping, or concurrent, experiments.<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a></li> <li>Robots, whether <a href="/facts/Web_crawlers/ra7D1xyb">web crawlers</a> from valid sources or malicious <a href="/facts/Internet_bots/yPTtRLyq">internet bots</a>.</li> <li>Ability to ramp-up experiments from low percentages to higher percentages.</li> <li>Speed / performance has significant impact on key metrics.<a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a><a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a></li></ul> <ul><li>Ability to use the pre-experiment period as an A/A test to reduce variance.<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a></li></ul> <h2 id="history">History</h2> <p class="note">Main article: <a href="/facts/History_of_experiments/QT89Nr88">History of experiments</a></p> <p>A controlled experiment appears to have been suggested in the Old Testament's <a href="/facts/Book_of_Daniel/m22qUIWF">Book of Daniel</a>. King Nebuchadnezzar proposed that some Israelites eat "a daily amount of food and wine from the king's table." Daniel preferred a <a href="/facts/Vegetarian_cuisine/303WNHkK">vegetarian</a> diet, but the official was concerned that the king would "see you looking worse than the other young men your age? The king would then have my head because of you." Daniel then proposed the following controlled experiment: "Test your servants for ten days. Give us nothing but vegetables to eat and water to drink. Then compare our appearance with that of the young men who eat the royal food, and treat your servants in accordance with what you see". (Daniel 1, 12– 13).<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a><a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> </p><p>Randomized experiments were institutionalized in psychology and education in the late eighteen-hundreds, following the invention of randomized experiments by <a href="/facts/Charles_Sanders_Peirce/z76dNqhM">C. S. Peirce</a>.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a><a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a><a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a><a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> Outside of psychology and education, randomized experiments were popularized by <a href="/facts/R.A._Fisher/I2FVIpy9">R.A. Fisher</a> in his book <i><a href="/facts/Statistical_Methods_for_Research_Workers/aRppN873">Statistical Methods for Research Workers</a></i>, which also introduced additional principles of experimental design. </p> <h2 id="statistical-interpretation">Statistical interpretation</h2> <p>The <a href="/facts/Rubin_Causal_Model/m6h998gw">Rubin Causal Model</a> provides a common way to describe a randomized experiment. While the Rubin Causal Model provides a framework for defining the causal parameters (i.e., the effects of a randomized treatment on an outcome), the analysis of experiments can take a number of forms. The model assumes that there are two potential outcomes for each unit in the study: the outcome if the unit receives the treatment and the outcome if the unit does not receive the treatment. The difference between these two potential outcomes is known as the treatment effect, which is the causal effect of the treatment on the outcome. Most commonly, randomized experiments are analyzed using <a href="/facts/ANOVA/HHR2jkDk">ANOVA</a>, <a href="/facts/Student%2527s_t-test/dMH8uC28">student's t-test</a>, <a href="/facts/Regression_analysis/n6z5Tf7K">regression analysis</a>, or a similar <a href="/facts/Statistical_hypothesis_testing/yv9RPF6U">statistical test</a>. The model also accounts for potential confounding factors, which are factors that could affect both the treatment and the outcome. By controlling for these confounding factors, the model helps to ensure that any observed treatment effect is truly causal and not simply the result of other factors that are correlated with both the treatment and the outcome. </p><p>The Rubin Causal Model is a useful a framework for understanding how to estimate the causal effect of the treatment, even when there are confounding variables that may affect the outcome. This model specifies that the causal effect of the treatment is the difference in the outcomes that would have been observed for each individual if they had received the treatment and if they had not received the treatment. In practice, it is not possible to observe both potential outcomes for the same individual, so statistical methods are used to estimate the causal effect using data from the experiment. </p> <h2 id="empirical-evidence-that-randomization-makes-a-difference">Empirical evidence that randomization makes a difference</h2> <p>Empirically differences between randomized and non-randomized studies,<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a>[<i>needs update</i>] and between adequately and inadequately randomized trials have been difficult to detect.<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a><a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> </p> <h2 id="directed-acyclic-graph-dag-explanation-of-randomization">Directed acyclic graph (DAG) explanation of randomization</h2> <p>Randomization is the cornerstone of many scientific claims. To randomize, means that we can eliminate the confounding factors. Say we study the effect of A on B. Yet, there are many unobservables U that potentially affect B and confound our estimate of the finding. To explain these kinds of issues, statisticians or econometricians nowadays use <a href="/facts/Directed_acyclic_graph/k6zq1os9">directed acyclic graph</a>.[<i>needs update</i>] </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/A%2fB_testing/0oDTfuhk">A/B testing</a></li> <li><a href="/facts/Allocation_concealment/W7kMhoqm">Allocation concealment</a></li> <li><a href="/facts/Random_assignment/TDeSKXtu">Random assignment</a></li> <li><a href="/facts/Randomized_block_design/EAwgBo1y">Randomized block design</a></li> <li><a href="/facts/Randomized_controlled_trial/LG9lkyVx">Randomized controlled trial</a></li></ul> <ul><li>Caliński, Tadeusz & Kageyama, Sanpei (2000). <a href="https://archive.org/details/blockdesignsrand0002cali"><i>Block designs: A Randomization approach, Volume I: Analysis</i></a>. Lecture Notes in Statistics. Vol. 150. New York: Springer-Verlag. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-387-98578-7.</li> <li>Caliński, Tadeusz & Kageyama, Sanpei (2003). <a href="https://archive.org/details/blockdesignsrand0002cali"><i>Block designs: A Randomization approach, Volume II: Design</i></a>. Lecture Notes in Statistics. Vol. 170. New York: Springer-Verlag. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-387-95470-7.</li> <li><a href="/facts/Ian_Hacking/qDHCULUy">Hacking, Ian</a> (September 1988). "Telepathy: Origins of Randomization in Experimental Design". <i><a href="/facts/Isis_(journal)/vqatJzvj">Isis</a></i>. 79 (3): 427–451. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1086%2F354775">10.1086/354775</a>. <a href="/facts/JSTOR_(identifier)/YTeVmaJ7">JSTOR</a> <a href="https://www.jstor.org/stable/234674">234674</a>. <a href="/facts/MR_(identifier)/uP137L11">MR</a> <a href="https://mathscinet.ams.org/mathscinet-getitem?mr=1013489">1013489</a>. <a href="/facts/S2CID_(identifier)/ldJsHa2Y">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:52201011">52201011</a>.</li> <li>Hinkelmann, Klaus; <a href="/facts/Oscar_Kempthorne/k9beaHYS">Kempthorne, Oscar</a> (2008). <a href="https://books.google.com/books?id=T3wWj2kVYZgC"><i>Design and Analysis of Experiments, Volume I: Introduction to Experimental Design</i></a> (Second ed.). Wiley. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-471-72756-9. <a href="/facts/MR_(identifier)/uP137L11">MR</a> <a href="https://mathscinet.ams.org/mathscinet-getitem?mr=2363107">2363107</a>.</li> <li><a href="/facts/Oscar_Kempthorne/k9beaHYS">Kempthorne, Oscar</a> (1992). <a href="http://projecteuclid.org/euclid.lnms/1215458836">"Intervention experiments, randomization and inference"</a>. In Malay Ghosh and Pramod K. Pathak (ed.). <i>Current Issues in Statistical Inference—Essays in Honor of D. Basu</i>. Institute of Mathematical Statistics Lecture Notes - Monograph Series. Hayward, CA: Institute for Mathematical Statistics. pp. 13–31. <a href="/facts/Doi_(identifier)/muM9Etpq">doi</a>:<a href="https://doi.org/10.1214%2Flnms%2F1215458836">10.1214/lnms/1215458836</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-940600-24-9. <a href="/facts/MR_(identifier)/uP137L11">MR</a> <a href="https://mathscinet.ams.org/mathscinet-getitem?mr=1194407">1194407</a>.</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Kohavi, Ron; Longbotham, Roger (2015). "Online Controlled Experiments and A/B Tests" (PDF). In Sammut, Claude; Webb, Geoff (eds.). Encyclopedia of Machine Learning and Data Mining. Springer. pp. to appear. <a href="http://www.exp-platform.com/Documents/2015%20Online%20Controlled%20Experiments_EncyclopediaOfMLDM.pdf" target="_blank">http://www.exp-platform.com/Documents/2015%20Online%20Controlled%20Experiments_EncyclopediaOfMLDM.pdf</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Kohavi, Ron; Longbotham, Roger; Sommerfield, Dan; Henne, Randal M. (2009). "Controlled experiments on the web: survey and practical guide". Data Mining and Knowledge Discovery. 18 (1): 140–181. doi:10.1007/s10618-008-0114-1. ISSN 1384-5810. <a href="https://doi.org/10.1007%2Fs10618-008-0114-1" target="_blank">https://doi.org/10.1007%2Fs10618-008-0114-1</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Kohavi, Ron; Longbotham, Roger; Sommerfield, Dan; Henne, Randal M. (2009). "Controlled experiments on the web: survey and practical guide". Data Mining and Knowledge Discovery. 18 (1): 140–181. doi:10.1007/s10618-008-0114-1. ISSN 1384-5810. <a href="https://doi.org/10.1007%2Fs10618-008-0114-1" target="_blank">https://doi.org/10.1007%2Fs10618-008-0114-1</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Kohavi, Ron; Deng, Alex; Frasca, Brian; Longbotham, Roger; Walker, Toby; Xu Ya (2012). "Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained". Proceedings of the 18th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. <a href="http://www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx" target="_blank">http://www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Kohavi, Ron; Deng Alex; Frasca Brian; Walker Toby; Xu Ya; Nils Pohlmann (2013). "Online controlled experiments at large scale". Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. Vol. 19. Chicago, Illinois, USA: ACM. pp. 1168–1176. doi:10.1145/2487575.2488217. ISBN 9781450321747. S2CID 13224883. <a href="9781450321747" target="_blank">9781450321747</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Kohavi, Ron; Longbotham, Roger; Sommerfield, Dan; Henne, Randal M. (2009). "Controlled experiments on the web: survey and practical guide". Data Mining and Knowledge Discovery. 18 (1): 140–181. doi:10.1007/s10618-008-0114-1. ISSN 1384-5810. <a href="https://doi.org/10.1007%2Fs10618-008-0114-1" target="_blank">https://doi.org/10.1007%2Fs10618-008-0114-1</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Kohavi, Ron; Deng Alex; Longbotham Roger; Xu Ya (2014). "Seven rules of thumb for web site experimenters". Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. Vol. 20. New York, New York, USA: ACM. pp. 1857–1866. doi:10.1145/2623330.2623341. ISBN 9781450329569. S2CID 207214362. <a href="9781450329569" target="_blank">9781450329569</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Deng, Alex; Xu, Ya; Kohavi, Ron; Walker, Toby (2013). "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data". WSDM 2013: Sixth ACM International Conference on Web Search and Data Mining. <a href="http://www.exp-platform.com/Pages/CUPED.aspx" target="_blank">http://www.exp-platform.com/Pages/CUPED.aspx</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Neuhauser, D; Diaz, M (2004). "Daniel: using the Bible to teach quality improvement methods". Quality and Safety in Health Care. 13 (2): 153–155. doi:10.1136/qshc.2003.009480. PMC 1743807. PMID 15069225. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1743807" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1743807</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Angrist, Joshua; Pischke Jörn-Steffen (2014). Mastering 'Metrics: The Path from Cause to Effect. Princeton University Press. p. 31. <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>Charles Sanders Peirce and Joseph Jastrow (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences. 3: 73–83. http://psychclassics.yorku.ca/Peirce/small-diffs.htm <a href="/wiki/Charles_Sanders_Peirce" target="_blank">/wiki/Charles_Sanders_Peirce</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>Hacking, Ian (September 1988). "Telepathy: Origins of Randomization in Experimental Design". Isis. 79 (3): 427–451. doi:10.1086/354775. JSTOR 234674. MR 1013489. S2CID 52201011. <a href="/wiki/Ian_Hacking" target="_blank">/wiki/Ian_Hacking</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Stephen M. Stigler (November 1992). "A Historical View of Statistical Concepts in Psychology and Educational Research". American Journal of Education. 101 (1): 60–70. doi:10.1086/444032. S2CID 143685203. <a href="/wiki/Stephen_M._Stigler" target="_blank">/wiki/Stephen_M._Stigler</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>Trudy Dehue (December 1997). "Deception, Efficiency, and Random Groups: Psychology and the Gradual Origination of the Random Group Design" (PDF). Isis. 88 (4): 653–673. doi:10.1086/383850. PMID 9519574. S2CID 23526321. <a href="https://pure.rug.nl/ws/files/71855616/237831.pdf" target="_blank">https://pure.rug.nl/ws/files/71855616/237831.pdf</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>Anglemyer A, Horvath HT, Bero L (April 2014). "Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials". Cochrane Database Syst Rev. 2014 (4): MR000034. doi:10.1002/14651858.MR000034.pub2. PMC 8191367. PMID 24782322. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8191367" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8191367</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>Odgaard-Jensen J, Vist G, et al. (April 2011). "Randomisation to protect against selection bias in healthcare trials". Cochrane Database Syst Rev. 2015 (4): MR000012. doi:10.1002/14651858.MR000012.pub3. PMC 7150228. PMID 21491415. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7150228" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7150228</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>Howick J, Mebius A (2014). "In search of justification for the unpredictability paradox". Trials. 15: 480. doi:10.1186/1745-6215-15-480. PMC 4295227. PMID 25490908. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4295227" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4295227</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> </ol>