Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Cognitive reflection test
open-in-new
<h2 id="basis-of-test">Basis of test</h2> <p>According to Frederick, there are two general types of cognitive activity called "system 1" and "system 2" (these terms have been first used by Keith Stanovich and Richard West<a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a>). System 1 is executed quickly without reflection, while system 2 requires conscious thought and effort. The cognitive reflection test has three questions that each have an obvious but incorrect response given by system 1. The correct response requires the activation of system 2. For system 2 to be activated, a person must note that their first answer is incorrect, which requires reflection on their own cognition.<a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a> </p> <h2 id="correlating-measures">Correlating measures</h2> <p>The test has been found to correlate with many measures of economic thinking, such as numeracy,<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> <a href="/facts/Temporal_discounting/MeSQfVHB">temporal discounting</a>, risk preference, and gambling preference.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> It has also been correlated with measures of <a href="/facts/Heuristics_in_judgement_and_decision_making/gwyv5AkX">mental heuristics</a>, such as the <a href="/facts/Gambler%27s_fallacy/UXV5j7dD">gambler's fallacy</a>, understanding of <a href="/facts/Regression_to_the_mean/l7hDWAVR">regression to the mean</a>, the <a href="/facts/Sunk_cost_fallacy/S0N2pv1C">sunk cost fallacy</a>, and others.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a><a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a><a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> </p><p><a href="/facts/Keith_Stanovich/IxjG1g5J">Keith Stanovich</a> found that cognitive ability is not strongly correlated with CRT scores because it will only lead to better CRT performance under certain conditions. First, the test-taker must recognize the need to override their system 1 response, and then they must have available cognitive resources to carry out the override. If the test-taker does not need to inhibit system 1 for the override, then the system 2 response immediately follows. Otherwise, they must have the capacity to sustain inhibition of system 1 in order to engage the system 2 response.<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> Contrarily, some researchers have assessed the validity of the assessment, using an advanced <a href="/facts/Item_response_theory/4WmQgExu">item response theory</a> method, and found that the CRT likely measures <a href="/facts/Intelligence/pRmAeRsw">cognitive ability</a>.<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> The authors of the study explain the validity of the CRT has been questioned due to the lack of <a href="/facts/Validity_(statistics)/9LNgtTUG">validity</a> studies and the lack of a <a href="/facts/Psychometrics/56X26MDI">psychometric</a> approach. </p> <h2 id="test-questions-and-answers">Test questions and answers</h2> <p>The original test penned by Dr. Frederick contained only the three following questions:<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> </p> <ol><li>A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?</li> <li>If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?</li> <li>In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?</li></ol> <p>The intuitive answers to these questions that "system 1" gives typically are: 10 cents, 100 minutes, and 24 days; while the correct solutions are: 5 cents, 5 minutes, and 47 days. </p> <h2 id="limitations-and-alternatives">Limitations and alternatives</h2> <p>Studies have estimated that between 44 and 51% of research participants have previously been exposed to the CRT.<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a><a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a> Those participants that are familiar with the CRT tend to outscore those with no previous exposure, which raises questions about the validity of the measure in this population.<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a><a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a> In an effort to combat limitations associated with familiarity, researchers have developed a variety of alternative measures of cognitive reflection.<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a><a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a><a class="footnote-ref" id="fnref:24" href="#fn:24"><sup>24</sup></a> Recent research, however, suggests that the CRT is robust to multiple exposure, so that despite the raw score increases in experienced participants, its correlations with other variables remain unaffected.<a class="footnote-ref" id="fnref:25" href="#fn:25"><sup>25</sup></a> </p><p>Another limitation is due to a lack of strong psychometric properties and scarcity of validity studies in the literature.<a class="footnote-ref" id="fnref:26" href="#fn:26"><sup>26</sup></a> The CRT was not designed in a manner that aligns with standards of the industry such as the <a href="/facts/Standards_for_Educational_and_Psychological_Testing/SuNTyGjx">Standards for Educational and Psychological Testing</a> which was developed by the <a href="/facts/American_Educational_Research_Association/M0aYS2jW">American Educational Research Association</a>, <a href="/facts/American_Psychological_Association/907fEfuc">American Psychological Association</a>, & <a href="/facts/National_Council_on_Measurement_in_Education/awbuzJ1T">National Council on Measurement in Education</a>. </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Dual_process_theory/3Dqz1r0K">Dual process theory</a>, it gives more background about "system 1" and "system 2" thinking</li> <li><i><a href="/facts/Thinking,_Fast_and_Slow/nnzRqhWL">Thinking, Fast and Slow</a></i>, book by Daniel Kahneman</li> <li><a href="/facts/Trick_question/Q3mDM5JN">Trick question</a></li> <li><a href="/facts/Wason_selection_task/HUK1jqx9">Wason selection task</a>, a test of deductive reasoning</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN 1099-0771. S2CID 197706996. <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133" target="_blank">https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi:10.1257/089533005775196732. <a href="https://doi.org/10.1257%2F089533005775196732" target="_blank">https://doi.org/10.1257%2F089533005775196732</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Oechssler, Jörg; Roider, Andreas; Schmitz, Patrick W. (2009). "Cognitive abilities and behavioral biases" (PDF). Journal of Economic Behavior & Organization. 72 (1): 147–152. doi:10.1016/j.jebo.2009.04.018. ISSN 0167-2681. <a href="https://epub.uni-regensburg.de/21701/2/roder2.pdf" target="_blank">https://epub.uni-regensburg.de/21701/2/roder2.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>Hoppe, Eva I.; Kusterer, David J. (2011). "Behavioral biases and cognitive reflection". Economics Letters. 110 (2): 97–100. doi:10.1016/j.econlet.2010.11.015. ISSN 0165-1765. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Toplak, Maggie (4 May 2011). "The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks" (PDF). Memory and Cognition. 39 (7): 1275–1289. doi:10.3758/s13421-011-0104-1. PMID 21541821. Retrieved 30 May 2014. <a href="http://keithstanovich.com/Site/Research_on_Reasoning_files/MC_2011.pdf" target="_blank">http://keithstanovich.com/Site/Research_on_Reasoning_files/MC_2011.pdf</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN 1099-0771. S2CID 197706996. <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133" target="_blank">https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Szaszi, B., Szollosi, A., Palfi, B., Aczél B., (2017) The cognitive reflection test revisited: exploring the ways individuals solve the test, Thinking and Reasoning, https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954 <a href="https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954" target="_blank">https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. https://doi.org/10.1017/s0140525x00003435 <a href="https://doi.org/10.1017/s0140525x00003435" target="_blank">https://doi.org/10.1017/s0140525x00003435</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi:10.1257/089533005775196732. <a href="https://doi.org/10.1257%2F089533005775196732" target="_blank">https://doi.org/10.1257%2F089533005775196732</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Szaszi, B., Szollosi, A., Palfi, B., Aczél B., (2017) The cognitive reflection test revisited: exploring the ways individuals solve the test, Thinking and Reasoning, https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954 <a href="https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954" target="_blank">https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi:10.1257/089533005775196732. <a href="https://doi.org/10.1257%2F089533005775196732" target="_blank">https://doi.org/10.1257%2F089533005775196732</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>Oechssler, Jörg; Roider, Andreas; Schmitz, Patrick W. (2009). "Cognitive abilities and behavioral biases" (PDF). Journal of Economic Behavior & Organization. 72 (1): 147–152. doi:10.1016/j.jebo.2009.04.018. ISSN 0167-2681. <a href="https://epub.uni-regensburg.de/21701/2/roder2.pdf" target="_blank">https://epub.uni-regensburg.de/21701/2/roder2.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Hoppe, Eva I.; Kusterer, David J. (2011). "Behavioral biases and cognitive reflection". Economics Letters. 110 (2): 97–100. doi:10.1016/j.econlet.2010.11.015. ISSN 0165-1765. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>Toplak, Maggie (4 May 2011). "The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks" (PDF). Memory and Cognition. 39 (7): 1275–1289. doi:10.3758/s13421-011-0104-1. PMID 21541821. Retrieved 30 May 2014. <a href="http://keithstanovich.com/Site/Research_on_Reasoning_files/MC_2011.pdf" target="_blank">http://keithstanovich.com/Site/Research_on_Reasoning_files/MC_2011.pdf</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>Stanovich, Keith E., & West, Richard F. (2008). "On the relative independence of thinking biases and cognitive ability." Personality Processes and Individual Differences, 94(4), 672-695. doi:10.1037/0022-3514.94.4.672 http://psycnet.apa.org/journals/psp/94/4/672 <a href="http://psycnet.apa.org/journals/psp/94/4/672" target="_blank">http://psycnet.apa.org/journals/psp/94/4/672</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN 1099-0771. S2CID 197706996. <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133" target="_blank">https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi:10.1257/089533005775196732. <a href="https://doi.org/10.1257%2F089533005775196732" target="_blank">https://doi.org/10.1257%2F089533005775196732</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>Haigh, Matthew (2016). "Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success?". Advances in Cognitive Psychology. 12 (3): 145–149. doi:10.5709/acp-0193-5. PMC 5225989. PMID 28115997. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5225989" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5225989</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>Stieger, Stefan; Reips, Ulf-Dietrich (2016-09-06). "A limitation of the Cognitive Reflection Test: familiarity". PeerJ. 4: e2395. doi:10.7717/peerj.2395. ISSN 2167-8359. PMC 5018679. PMID 27651989. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5018679" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5018679</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>Haigh, Matthew (2016). "Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success?". Advances in Cognitive Psychology. 12 (3): 145–149. doi:10.5709/acp-0193-5. PMC 5225989. PMID 28115997. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5225989" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5225989</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>Stieger, Stefan; Reips, Ulf-Dietrich (2016-09-06). "A limitation of the Cognitive Reflection Test: familiarity". PeerJ. 4: e2395. doi:10.7717/peerj.2395. ISSN 2167-8359. PMC 5018679. PMID 27651989. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5018679" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5018679</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>Primi, Caterina; Morsanyi, Kinga; Chiesi, Francesca; Donati, Maria Anna; Hamilton, Jayne (2016-12-01). "The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)". Journal of Behavioral Decision Making. 29 (5): 453–469. doi:10.1002/bdm.1883. hdl:2158/1011727. ISSN 1099-0771. S2CID 56252490. <a href="https://qub.elsevierpure.com/en/publications/0063c34c-74cc-4ad8-b844-674bd38db96b" target="_blank">https://qub.elsevierpure.com/en/publications/0063c34c-74cc-4ad8-b844-674bd38db96b</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>Toplak, Maggie E.; West, Richard F.; Stanovich, Keith E. (2014-04-03). "Assessing miserly information processing: An expansion of the Cognitive Reflection Test". Thinking & Reasoning. 20 (2): 147–168. doi:10.1080/13546783.2013.844729. ISSN 1354-6783. S2CID 53340418. <a href="/wiki/Doi_(identifier)" target="_blank">/wiki/Doi_(identifier)</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> <li id="fn:24"><p>Thomson, Keela S.; Oppenheimer, Daniel M. (2016). "Investigating an alternate form of the cognitive reflection test". Judgment and Decision Making. 11: 99–113. doi:10.1017/S1930297500007622. S2CID 146924609. <a href="http://journal.sjdm.org/15/151029/jdm151029.html" target="_blank">http://journal.sjdm.org/15/151029/jdm151029.html</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></p></li> <li id="fn:25"><p>Bialek, Michal; Pennycook, Gordon (2017-08-28). "The Cognitive Reflection Test is robust to multiple exposures". Behavior Research Methods. 50 (5): 1953–1959. doi:10.3758/s13428-017-0963-x. PMID 28849403. <a href="/wiki/Gordon_Pennycook" target="_blank">/wiki/Gordon_Pennycook</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></p></li> <li id="fn:26"><p>Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN 1099-0771. S2CID 197706996. <a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133" target="_blank">https://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.2133</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></p></li> </ol>