{"id":3798,"date":"2020-12-09T14:10:57","date_gmt":"2020-12-09T13:10:57","guid":{"rendered":"http:\/\/blogs.kcl.ac.uk\/editlab\/?p=3798"},"modified":"2021-01-18T11:28:13","modified_gmt":"2021-01-18T10:28:13","slug":"self-report-measures-and-the-replication-crisis","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/editlab\/2020\/12\/09\/self-report-measures-and-the-replication-crisis\/","title":{"rendered":"Self-Report Measures and the Replication Crisis"},"content":{"rendered":"

In our next EDIT Lab blog installment, Zain discusses the replication crisis facing the field of psychiatry and how self-reported measures may, in part, be a contributing factor.<\/h2>\n

 <\/p>\n

\n

 <\/p>\n

It is a widely accepted fact that the \u2018replication crisis\u2019 exists in psychiatry. The replication crisis refers to a situation where the results from studies prove difficult or impossible to replicate in subsequent attempts to repeat the same experiment (Leichsenring et al. 2017). This has been a topic of much discussion since the 1990s, but methodologies still require work to be more reproducible. There are numerous reasons why a set of results fail to be replicated, and among these is the heterogeneity of self-reported measures often relied upon in studies.<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

In psychiatry, self-reported measures are usually contrasted with clinician-reported measures. In self-reported measures, the patient will complete questions and provide information about their own symptoms and feelings, whereas with clinician-reported, the clinician will interview the patient and use these to inform their report of the patients\u2019 symptoms and feelings. Clinician-reported measures are subject to their own biases, but generally are more reliable than self-reported measures. One of the biggest confounders in self-reported measures is social desirability bias.<\/span><\/p>\n

Patients have a tendency to give answers in self-reported questionnaires which present a favourable image of themselves. This is referred to as \u2018socially desirable responding\u2019 (SDR) (Paulhus 2002). Numerous measures are subject to social desirability, including income, illicit drug use, alcohol consumption and feelings of low mood. This leads to self-reported measures giving an inaccurate picture of the patient\u2019s true situation, where \u2018socially undesirable\u2019 traits and symptoms are diminished, and \u2018socially desirable\u2019 traits are enhanced. The fact that this bias is not uniform and affects individuals, samples and questions differently significantly complicates the later replication attempts. All is not lost, however.<\/span><\/p>\n

Yang et al. showed in a 2020 paper that it was possible to correct, at least partially, for inaccuracies in alcohol consumption measures from the UK Biobank sample by using other measures and inconsistencies across measures to deduce when patient responses to alcohol consumption questions were not accurate. For example, patients with a history of alcoholic liver disease who had listed themselves as \u2018never drinkers\u2019 were possibly demonstrating social desirability and not being completely truthful in their answers.\u00a0<\/span><\/p>\n

The application of this method actually resulted in the \u2018J-shaped curve\u2019 relationship between alcohol consumption and disease risk being modified. Where previously results had made it appear as if low alcohol consumption was actually beneficial for health, it turned out that this was due to mis-categorisation of these patients. In reality, no amount of alcohol is beneficial for health. There is hope for a similar approach in measures of other diseases too.<\/span><\/p>\n

For example, with low mood and depressive symptoms, an experimenter could review a patient\u2019s drug history and categorise patients who have historically been prescribed antidepressants as having a history of low mood, or the same for current low mood and current antidepressant prescription. Measures such as these would not eliminate the effects of social desirability, but would at least correct for it to some degree, and give researchers some insight into the size of the SDR effect in their sample.\u00a0<\/span><\/p>\n

As well as acknowledging that the replication crisis exists, researchers might also benefit from appreciating that social desirability partly contributes to it. We can bear this idea in mind when designing our studies and we attempt to maximise the possibility of reproducibility. In order to achieve optimal reproducibility, we need to continue to develop and use better measures of symptoms and traits. In studies where self-reported measures are the only feasible method of data collection, we must always ask ourselves \u2018How much can I trust what the patient is telling me and how can I verify it?\u2019<\/span><\/p>\n

\"\"<\/a><\/p>\n

References<\/b>\u00a0<\/span><\/p>\n

Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S. and Steinert, C. (2017) \u201cBiases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research,\u201d Psychological Medicine. Cambridge University Press, 47(6), pp. 1000\u20131011.<\/span><\/p>\n

Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (p. 49\u201369).<\/span><\/p>\n

Xue A, Jiang L, Zhu Z, Wray NR, Visscher PM, Zeng J, et al. Genome-wide analyses of behavioural traits biased by misreports and longitudinal changes. medRxiv [Internet]. 2020 Jan 1<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

In our next EDIT Lab blog installment, Zain discusses the replication crisis facing the field of psychiatry and how self-reported measures may, in part, be a contributing factor.     It is a widely accepted fact that the \u2018replication crisis\u2019 exists in psychiatry. The replication crisis refers to a situation…<\/p>\n","protected":false},"author":941,"featured_media":3832,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[87,43,290,230,349],"class_list":["post-3798","post","type-post","status-publish","format-standard","has-post-thumbnail","category-research-matters","tag-awareness","tag-mental-health","tag-methods","tag-replication","tag-self-reports"],"_links":{"self":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/3798","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/users\/941"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/comments?post=3798"}],"version-history":[{"count":7,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/3798\/revisions"}],"predecessor-version":[{"id":3859,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/posts\/3798\/revisions\/3859"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/media\/3832"}],"wp:attachment":[{"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/media?parent=3798"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/categories?post=3798"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/editlab\/wp-json\/wp\/v2\/tags?post=3798"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}