Brain imaging, the legal process and neurofantasy; can it be cured (by applying the principles of evidence – based medicine)?

Peter Sandercock * and Joanna Wardlaw**

1. Introduction

Until the latter part of the 20th century, the medical technology that was available to guide the diagnosis and treatment of people with brain disorders could only provide relatively crude information. In 1971, however, Hounsfield1 developed the computed tomographic scanner. It provided, for the first time, images of living brain tissue, and very soon after that became sufficiently reliable that it entered routine clinical practice. Since then, brain imaging techniques have advanced extremely rapidly, and clinicians now have very powerful tools such as magnetic resonance imaging (MRI). With techniques such as tractography and functional MRI (fMRI) they can locate and characterise, in great detail, disorders of brain structure, interconnectivity and function. These developments have enabled neuroscientists to explore many of the aspects of the function of the normal human brain that underlie cognition, memory and action. Coverage of these new developments, by both the scientists who promote their work and the media who report them, has not always presented a balanced picture of what the new research findings really mean. Caulfield et al expressed the problem succinctly:

The increasing popularity of neuroimaging studies among the research community in recent decades has also garnered interest from the media. But there is concern that coverage of controversial topics, such as the ability of fMRI to “read minds”, has sensationalised the neuroimaging field and led to mounting criticism and skepticism.2

Some of the wilder claims for the capabilities of neuroimaging have been described as ‘Neurofantasies’.3 This begs the question as to whether the technology (or society) is ready to be exposed to the implications of these wider non-medical uses.

2. The Brain Imaging and Society Programme4

Widespread application of these new techniques for very diverse purposes, well beyond the medical diagnostic purpose for which they were originally designed, may have all sorts of unintended consequences. “A picture speaks a thousand words” is as true in the scientific context as it is in literature. MRI images are a powerful way of conveying a message, often in excess of the validity of the finding that they represent, something that has not been lost on the media. An image of a brain, colour-highlighted to indicate the locus responsible for the individual’s political preferences,5 accompanied by the opinion of an eminent expert on the importance or relevance (but rarely the reliability) of the intriguing new research can evoke an “I told you so” response in the reader. It can be too beguiling however to enable a largely lay audience to assess the validity of the new research.

2.1 The need for critical appraisal of the scientific evidence

Edward Vul from MIT has undertaken a critical analysis of the statistical procedures that generate some of these very persuasive images of brain processes, raising concerns that some of the more striking findings may be no more than statistical artefacts, which he refers to as ‘voodoo correlations’.6 Scientific work yielding strongly positive results is much more likely to be published in a scientific journal, and hence become widely known to the general public, than studies with less striking results. There is good theoretical and empirical evidence, however, to show that the findings of strikingly positive initial studies may over-estimate the true effects under study and are often not replicated by later studies7,8

2.2 Viewing new scientific results in context – the need for research evidence synthesis

To assess the importance of a new scientific neuroimaging finding, one needs to know not only that the results of the experiment giving rise to it are reliable, but also how it should be interpreted in the context of results of all similar experiments. The situation is commonly encountered in many branches of medicine, where some medical findings (for example new treatments) may be compelling because they are consistent with the accepted dogma of the time, whereas other ideas are less well accepted because they appear to contradict current thinking or practice.

2.2.1 Individual Studies Too Small To Show the True Effect

The history of research into clot-busting treatments for heart attacks provides a good example. Most heart attacks are due to a blood clot blocking one of the main blood vessels to the heart. The clot cuts off the blood and oxygen supply to the heart muscle, which prevents it from working properly, and can lead to death. “Clot busting” (or thrombolytic) drugs, which dissolve the blood clot, reopen the artery and reduce damage to the heart muscle, are now in routine use worldwide. In the 1970s, however, when they were first introduced, the results of a few small clinical trials suggested that thrombolytic drugs were very dangerous. Textbooks and reviews written by opinion-leaders on the treatment of heart attacks over the next 15 years either made no mention of thrombolytic drugs, or indicated that they were experimental.9 In truth, by the early 1980s when those books and articles were being written, sufficient evidence had been accumulated from randomised controlled trials to demonstrate beyond reasonable doubt that clot busting treatment saved the lives of heart attack patients. This effect only became apparent, however, when a systematic and quantitative analysis (meta-analysis) of all available clinical trials was performed (see Figure). In this instance, none of the individual studies were large enough to have provided convincing evidence on their own. This and other similar meta- analyses subsequently stimulated research studies that were large enough to confirm this important medical benefit.10

Figure. Cumulative meta-analysis by year of publication of randomized control trials (RCTs) of thrombolytic therapy for myocardial infarction are presented on the left. The cumulative number of trials and patients (Pts) are also presented. On the right, the recommendations of the clinical expert reviewers are presented in 2-year segments, except for the entry in 1966, which represents all previous years. The letter M indicates that at least one meta-analysis was published that year; NS indicates not significant. (Reproduced with permission from the author and The American Medical Association).

2.2.2 Single Very Positive Research Study Must be Set in Context of Many ‘Negative’ Studies

Since the 1960s, blood thinning (anticoagulant) drugs have been widely used, despite the absence of reliable evidence from clinical trials, for the emergency treatment of patients with symptoms of acute stroke. A single strikingly positive – but relatively small – clinical trial published in 1995 appeared to support this practice.11 As with the previous example of clot-busting drugs for treatment of heart attacks, however, it is important that such results – however striking they might appear – are reviewed in the context of all of the evidence that is available. A systematic review of all the trials of anticoagulant drugs in patients with acute stroke showed that there was no net benefit from such anticoagulants. The apparent benefits seen in the 1995 study may have arisen by the play of chance, and may not represent the true effect of the drug.12

3. The need for the Brain Imaging and Society Programme

In the light of concerns about potential misuse (or at least misinterpretation) of the emerging advanced neuroimaging techniques, the Brain Imaging and Society Programme was established in 2010. The programme sought to stimulate debate among members of a community including academics, government officials, legal professionals, ethicists and the lay public, in order to understand how sections of society perceive and react to the implications of advanced neuroimaging techniques. The programme aims: firstly, to understand the context of the societal sectors in which neuroimaging is being used and is beginning to have an influence, particularly law and neuromarketing;13 secondly, to study the legal and human rights implications of these uses; and, thirdly, to consider what constitutes acceptable practice, and whether – and if so how – it should be regulated. In this article, we focus on implications for the use of neuroimaging in law, and how the application of some of the principles developed for critical appraisal of medical data (so called Evidence-Based Medicine) might aid a more balanced appreciation of the value of neuroimaging in a legal context.

4. Critical appraisal, evidence-based medicine (EBM) and neuroimaging

The idea that evidence (resulting from well-conducted research) is a more reliable guide to action than opinion is not new, but has been articulated increasingly clearly under various different guises, throughout the 20th century. Evidence should be as free of bias, as up to date, and as complete as possible, to enable users of that information to make the most well-informed decisions. In medical practice, David Sackett and his group at McMaster University, leading proponents of EBM, set out very clear principles for EBM that are as valid today as when they were published in 1985.14 For application to neuroimaging and law, the principles of EBM could be expressed as a series of questions, intended to assist the reader/user to determine the validity of neuroimaging results and their applicability to a given specific purpose:

• From what type of subject was the data collected (normal volunteer, patient with a known medical disorder, prisoner, suspected terrorist)?

• In what type of setting was the data collected (research laboratory, hospital, other facility?

• What was the purpose of the original research study (was the “research question” expressed clearly at the outset, was the study designed and conducted rigorously, and did it include enough subjects to answer the question reliably?).

• If the study was designed to predict future health events (ie prognosis), what type of people do the results apply to?

• If more than one study has examined a particular question, has the scientist made a rigorous search for all reports of such other studies, in order to avoid bias in the selection of the evidence for review?

• Have appropriate methods been used to synthesise the evidence from those studies (referred to as meta-analysis) to achieve a reliable overall view of the data?

4.1 The Cochrane Collaboration

The questions set out in the previous section, asked systematically of data from a given field of research, provide a useful tool for assessment of the overall strength of evidence, and hence the reliability of any conclusions based on those data. An application in the field of healthcare is the Cochrane Collaboration.15 The Collaboration is an international network established in 1993 to prepare, maintain and promote Cochrane Systematic Reviews of all available relevant evidence on specific forms of healthcare. These reviews, of which there are so far over 4,000 published online in the Cochrane Library,16 help healthcare providers, policymakers, patients, patient advocates and carers to make well-informed decisions. The reviews assembled by the Cochrane Collaboration are highly regarded, as they are perceived to be of high quality and to be as free of bias as is practicable. Initially, they were restricted to therapeutic interventions, but the Collaboration has recently broadened its scope to include reviews of the accuracy of diagnostic tests. This is particularly relevant to neuroimaging, since neuroimaging applications might be called upon to address questions of a diagnostic nature (eg “is this person a liar?”) in a legal setting. Although there are plans for systematic reviews of studies involving prognosis (eg “will this person offend again?”), methods for providing a quantitative synthesis of such evidence are not yet well developed.

4.2 The Campbell Collaboration

The evaluation of neuroimaging, when applied outside a healthcare context, requires a different perspective. The standard of evidence required to support the use of a particular neuroimaging technique in a criminal court, for example, would be very different to that required if the same technique were to be applied in a medical setting. It is fortunate therefore that the Campbell Collaboration,17 an organisation, founded on similar principles to the Cochrane Collaboration, produces systematic reviews of the effects of social interventions in areas such as education, crime and justice, and social welfare. Like the Cochrane, the Campbell Collaboration is based on voluntary cooperation among researchers of a variety of backgrounds. Transparency in the reporting of primary evidence is key to the production of high-quality systematic reviews.  Reviewers need enough information to be able to assess what constitutes the “best” evidence. A good example is a recent systematic review which cast doubt on the value of court-mandated interventions for individuals convicted of domestic violence. The findings were based on several studies which together included 3614 participants. The authors concluded:

The findings showed a small positive effect when official measures of repeat violence were examined but no effect when victim reports of repeat violence were used. The inconsistency in findings across measures and the greater credibility of the victim based data raise serious concerns about the effectiveness of these programs.18

4.3 The CONSORT statement

A systematic review of the available evidence, in addition to providing a summary estimate of the effect of a treatment (or accuracy of a diagnostic test) based on the totality of the evidence, often highlights limitations in the underlying individual clinical trials. Such limitations include the possibility that the trials did not include a sufficient number of patients, or did not include a relevant or representative cross-section of patients. The trials might have been well conducted but the description of the study methods was not adequate to enable the reader to judge the reliability of the evidence. These problems lead to the development of the CONSORT Statement,19 originally published in 1996, to address the issue of reporting transparency in healthcare trials.  It is now an established instrument that has been adapted internationally by journal editors and societal bodies. The CONSORT and Criminal Justice Trials (CJT)20 project, led by Campbell Crime & Justice Group member Dr Amanda Perry, has devised a coding sheet to enable application of the CONSORT statement to a range of crime and justice trials.  It requires, for example, that reports clearly state the primary method of assessment of the outcome of the research study (ie the results upon which the study will be judged to have confirmed or refuted the hypothesis under test). In a systematic review of interventions for the prevention of repeat domestic violence,21 the determination of outcome is clearly critical to findings on effectiveness of the intervention. If the primary outcome is “official measures of repeat violence”, intervention appears effective, but if it is “victim reports of repeat violence”, then perhaps not.

5. Conclusions

The communication of new scientific developments in neuroimaging, as in other fields, may be subject to distortions22 and misunderstandings. Consideration of the principles set out here, by the scientists who perform neuroimaging research and the media that report it to the general public, could reduce the risk of exaggerated claims leading to excessive scepticism. We hope that when reviewing new claims for neuroimaging – such as the reliability of a brain scan diagnosis of autism – at least some parts of the media might develop the healthy, but balanced (and evidence-based) scepticism displayed in newspaper columns such as Ben Goldacre’s ‘Bad Science’ in the Guardian.23 Recent examples include a study showing how easy it is to be misled as to the cause of witnessed events,24 and that the claims regarding a bomb detector were completely unfounded.25 We hope that, if scientists, the media, the legal system and the general public were to employ the principles of Evidence Based Medicine when appraising the evidence provided by neuroimaging reports, they would be able to respond to it intelligently, being neither too sceptical nor overly impressed by the hype.

---