{"id":2866,"date":"2018-10-17T10:24:17","date_gmt":"2018-10-17T09:24:17","guid":{"rendered":"http:\/\/blogs.kcl.ac.uk\/editlab\/?p=2866"},"modified":"2018-10-17T11:35:16","modified_gmt":"2018-10-17T10:35:16","slug":"s-for-statistical-significance","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/editlab\/2018\/10\/17\/s-for-statistical-significance\/","title":{"rendered":"S for Statistical Significance"},"content":{"rendered":"
\n

This week for our S blog, we bring a post on the important issue of statistical significance, written by a guest blogger from the Said & Dunn blog<\/a>, led by Dr Erin Dunn. This post is by Khalil Zlaoui, a graduate student in the Dunn Lab. We are very grateful to him for sharing this content with us. The standard measure of statistical significance in science (i.e. whether you believe a result is true) is when your effect is associated with a p value of less than 0.05. What this means is that you would expect that finding by chance no more than one in twenty times. Here Khalil discuss various problems with that approach and suggestions that have been made to address them.<\/strong><\/em>
\n<\/span><\/p>\n<\/div>\n

\"\"<\/a><\/p>\n

\n
\n
\n

 <\/p>\n

Scientific studies often begin with a hypothesis about the way the world works. The hope is that by analyzing data we can find empirical evidence to uncover the truth, by either supporting or refuting our original hypothesis.<\/p>\n

For decades, the scientific community has relied on the\u00a0p-value\u00a0<\/em>as the sole indicator of that truth.\u00a0\u00a0But a significant\u00a0p-value\u00a0<\/em>is not proof of strong evidence, and in fact, it was never intended to be used as such. Its misuse and misinterpretation has led to serious problems, including an interdisciplinary replication crisis that we describe in a previous\u00a0Said&Dunn post\u00a0<\/a>.<\/p>\n

“Should more journals ban\u00a0p-values\u00a0<\/em>altogether?”<\/p><\/blockquote>\n

Benjamin et al. (2017)<\/a>\u00a0recently made a proposal to change the default\u00a0p-value\u00a0<\/em>threshold (from 0.05 to 0.005) for claims of new discoveries, a proposal met with some\u00a0controversy<\/a>.\u00a0\u00a0Some journals, like\u00a0Basic and Applied Social Psychology<\/em>, have also gone so far as to ban the use of\u00a0p-values<\/em>.<\/p>\n

\"\"<\/a><\/p>\n

Is tightening the\u00a0p-value\u00a0<\/em>threshold a good idea? Should more journals ban\u00a0p-values\u00a0<\/em>altogether? After wading into the waters around the promises and pitfalls of\u00a0p-values<\/em>, here are four main conclusions we have drawn.<\/p>\n

1)\u00a0\u00a0<\/strong>P-VALUES\u00a0<\/em><\/strong>SHOULD BE INTERPRETED WITH CAUTION<\/strong><\/h3>\n

In 2016, the\u00a0American Statistical Association<\/a>\u00a0(ASA) argued that while the\u00a0p-value\u00a0<\/em>is a useful measure, it has been misused and misinterpreted in weighing evidence. The ASA board therefore released six principles to guard against common misconceptions of\u00a0p-values<\/em>:<\/p>\n

    \n
  1. P<\/em>-values\u00a0<\/em>can indicate how incompatible the data are with a specified statistical model.<\/strong>\u00a0\u00a0As models are constructed under a set of assumptions, a small\u00a0p-value\u00a0<\/em>indicates a model that is incompatible with the null hypothesis, as long as these assumptions hold.<\/li>\n
  2. A common misuse of\u00a0p-values\u00a0<\/em>is that they are often turned into statements about the truth of the null hypothesis.\u00a0P<\/em>-values\u00a0<\/em>do not measure the probability that the studied hypothesis is true.\u00a0\u00a0<\/strong>They also do not indicate the probability that data were produced by random chance alone.<\/li>\n
  3. Scientific conclusions and business or policy decisions should not be based only on whether a\u00a0p-value\u00a0<\/em>passes a specific threshold.\u00a0<\/strong>Conclusions based solely on\u00a0p-values\u00a0<\/em>can pose a threat to public-health. In addition to model design and estimation, factors to be considered in decision-making include study design and measurement quality.<\/li>\n
  4. \u00a0<\/strong>Proper inference requires full reporting and transparency.<\/strong>\u00a0\u00a0Conducting several tests of association in order to identify a significant\u00a0p-value\u00a0<\/em>leads to spurious results.<\/li>\n
  5. A\u00a0p<\/em>-value<\/em>, or statistical significance, does not measure the size of an effect or the importance of a result.\u00a0<\/strong>A smaller\u00a0p-value\u00a0<\/em>is not an indicator for a larger effect.<\/li>\n
  6. By itself, a\u00a0p<\/em>-value\u00a0<\/em>does not provide a good measure of evidence regarding a model or a hypothesis<\/strong>. A\u00a0p-value\u00a0<\/em>near 0.05 is only weak evidence against the null.<\/li>\n<\/ol>\n

    2) \u00a0<\/strong>THERE ARE ALTERNATIVES TO\u00a0P-VALUES<\/em><\/strong><\/h3>\n

    An interesting alternative to p-values is using a Bayesian approach, a method of statistical inference that includes a subjective \u201cprior\u201d belief about the hypothesis, based on Bayes theorem.<\/p>\n<\/div>\n

    \n

    From Bayes theorem:\u00a0POST<\/em>H1<\/sub>=PRIOR<\/em>H1<\/sub>\u00a0X\u00a0BF<\/em>, where\u00a0POST<\/em>\u00a0H1<\/sub>\u00a0is the posterior odds in favor of H1<\/sub>\u00a0(the alternative hypothesis<\/strong>) and\u00a0BF<\/em>\u00a0=\u00a0sampling density of data under H1<\/sub><\/ins>\u00a0 \u00a0\/ sampling density of data under H0\u00a0<\/sub>is the Bayes factor.\u00a0 \u00a0<\/span><\/p>\n<\/div>\n

    \n

    Interestingly, Bayes Factors can be equated with\u00a0p-values<\/em>. The correspondence between\u00a0p-values\u00a0<\/em>in the frequentist world (meaning the statistical inference framework that is most commonly used) and Bayes Factors in the Bayesian world can\u00a0reshape the debate<\/a>\u00a0about\u00a0p-values\u00a0<\/em>and help reconsider how strongly they can support evidence to reject the null.<\/p>\n

    Under some reasonable assumptions, a\u00a0p-value\u00a0<\/em>of 0.05 in the frequentist world corresponds to Bayes Factors in favor of the alternative hypothesis ranging from 2.5 to 3.4. In the Bayesian world, this is considered as weak evidence against the null. Based on the correspondence between p-values and Bayes Factors, Benjamin et al. (2017) proposed to\u00a0redefine statistical significance at 0.005<\/a>.<\/p>\n

    3) \u00a0<\/strong>TIGHTENING THE\u00a0P-VALUE\u00a0<\/em>THRESHOLD DOESN\u2019T FULLY SOLVE THE REPLICATION CRISIS AND COULD LEAD TO OTHER PROBLEMS<\/strong><\/h3>\n

    “p-values\u00a0<\/em>are not the only cause for the lack of reproducibility in science”<\/p><\/blockquote>\n

    A two-sided 0.005\u00a0p-value\u00a0<\/em>corresponds to Bayes Factors in favor of the alternative ranging from 14 to 26, which in Bayesian considerations corresponds to substantial to strong evidence.\u00a0\u00a0Benjamin et al.\u2019s proposal to change the p-value threshold was made to help address the replication crisis, where too few studies were able to replicate the findings of the original study.\u00a0\u00a0But moving to this more stringent threshold comes at a price \u2013 the need for larger samples and possibly unacceptable false negative rates.\u00a0\u00a0Is such a trade-off worth it?<\/p>\n

    It\u2019s important to keep in mind that\u00a0p-values\u00a0<\/em>are not the only cause for the lack of reproducibility in science.\u00a0\u00a0While\u00a0p-values\u00a0<\/em>might be an important contributor, there are other\u00a0real issues affecting replication<\/a>, including: selection effects, trends towards multiple testing, hunting for significance or p-hacking, violated statistical assumptions, and so on. Tightening the p-value from 0.05 to 0.005 will not necessarily address these issues.<\/p>\n

    4) \u00a0BUT THERE ARE ALTERNATIVES WE SHOULD ALL BE USING<\/strong><\/h3>\n

    Some statisticians have argued in favor of estimation (putting emphasis on the parameter to estimate an effect) over testing (putting emphasis on rejecting or accepting a hypothesis based on\u00a0p-values<\/em>). If the interest lies in testing an effect, researchers could instead rely on confidence, credibility, or prediction intervals.<\/p>\n

    So in the end, we think that rather than ditch the p-value altogether, we should shift our focus from\u00a0p-values\u00a0<\/em>to study design, effect size and confidence intervals, which we hope can help us better understand the evidence to support our hypotheses and ultimately uncover the truth about the way the world works.<\/p>\n

     <\/p>\n<\/div>\n

    \n
    \n
    \n
    \n

    \n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    This week for our S blog, we bring a post on the important issue of statistical significance, written by a guest blogger from the Said & Dunn blog, led by Dr Erin Dunn. This post is by Khalil Zlaoui, a graduate student in the Dunn Lab. We are very grateful…<\/p>\n","protected":false},"author":218,"featured_media":2868,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[248],"tags":[290,230,78,219],"class_list":{"0":"post-2866","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-a-z","8":"tag-methods","9":"tag-replication","10":"tag-research","11":"tag-statistics"},"_links":{"self":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/2866","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/users\/218"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/comments?post=2866"}],"version-history":[{"count":6,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/2866\/revisions"}],"predecessor-version":[{"id":2874,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/2866\/revisions\/2874"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/media\/2868"}],"wp:attachment":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/media?parent=2866"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/categories?post=2866"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/tags?post=2866"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}