DON'T GET TOO EXCITED: THE EFFECT MAY WEAR OFF!
This paper appeared in the Winter 2012 issue of the Skeptical Adversaria, the newsletter of ASKE, the Association for Skeptical Enquiry.
Many years ago, in my role as an NHS clinical psychologist, I had to see a man with a form of agoraphobia who, because of his panic attacks, could only walk unescorted to places within a short distance from his home. My method of helping him was simply to teach him ways of controlling his anxiety, to walk with him from his home, increasing the distance each time, and to encourage him to do this on his own. On our first excursion, he told me that when he first had the symptoms of agoraphobia he had his own one-man painting and decorating business but he was able to continue working because he felt safe when he was in his van and when he was somewhere where he had easy access to it should he start panicking. However, his van broke down, he could not afford the repairs and he had to take it off the road when the tax ran out. Hence he had to stop work He believed that if he could get it repaired and taxed or buy a new van he would be able to resume working and build up his business again. This in itself would be of major benefit for his confidence to overcome his anxiety, which had understandably been severely eroded by his not being able to work.
One problem I have found working with agoraphobic patients in this way is that often their lifestyle is such that there are few reasons or incentives for their going out on their own in the first place. This was certainly true of the above person and it struck me at the time that, instead of the NHS paying me lots of money to treat him, perhaps the money could have been better spent on repairing and taxing his van so he could resume work (and stop claiming benefits).
This was of course I pipe dream, which I immediately dismissed from my mind. However, my thoughts returned to this man with the recent publication by the Department of Health of a pilot study of 'personal health budgeting' begun in 2009 (note 1). This is a scheme for patients suffering from long-term medical conditions in England. A care plan is devised by the patient and the primary care trust that sets out the person's health needs, the amount of money available to meet those needs, and how this money will be spent.
The evaluation compared the experiences of people selected to receive personal health budgets with those of people continuing with conventional arrangements. The main findings were as follows:
- The use of personal health budgets was associated with a significant improvement in the care-related quality of life (the ASCOT-Adult Social Care Outcomes Toolkit), likewise psychological well?being as measured by the GHQ-12 (at 90% confidence). The GHQ-12 (General Health Questionnaire-12) is a list of 12 items relating to everyday mood, stress level, emotional wellbeing, etc., each of which the respondent rates).
- Personal health budgets did not appear to have an impact on health status per se over the 12-month follow?up period. No significant effects were found with regard to two clinical measures (HbA1C and lung?function tests, used where relevant) and there was no significant difference in mortality rates between the groups. Consistent with these results, the study did not find that personal health budgets had a significant effect on the EQ?5D compared to the control group. (The HbA1C is a blood test to check that diabetes is under control; the EQ?5D is a questionnaire completed by patients about their current physical and mental health.)
- Cost benefits were generally found to be favourable for personal health budgets (the analysis of this is rather complex).
- Patients who received personal health budgets were generally positive about the scheme.
In response to the report, social care minister Norman Lamb announced that the government will invest £1.5 million to hand over personal budgets to the estimated 56,000 people in receipt of NHS Continuing Care. The rollout will be completed by 2014. It is estimated that £90 million could be saved if just half of patients eligible for NHS Continuing Healthcare sign up to the programme.
Patients involved in the pilot scheme typically chose to spend the money on facilities such as carers and aids to mobility. However some opted for less obvious items. One gentleman, who was recovering from a stroke, was reported in the media to have spent his money on a drum kit and a sat-nav for his car, and a patient with motor neurone disease used his personal budget for a modified bicycle and membership of a gym. Other headlines spoke of singing lessons and manicures.
Needless to say this has provoked some expressions of outrage, and sceptics collecting specimens of prejudiced and bigoted thinking may wish to download the comments of some readers of The Telegraph in response to that newspapers article on the topic (note 2). That aside, and notwithstanding the attraction for sceptics of this kind of radical, thinking-outside-of-the-box approach, I have my own doubts. Firstly, changes in scores on rating scales and self-report questionnaires that measure very general conditions such as wellbeing and quality of life seem to me to provide a very weak basis for drawing any robust conclusions. Secondly, over the years I (and I imagine many others) have so often listened to or read the announcement that some new initiative 'is expected to save the NHS millions of pounds'. Surely if all of these initiatives had achieved this promise, the NHS would not be so constantly strapped for cash while forever devouring billions more?
My suspicion is that when the personal health budgeting scheme is rolled out it will become more expensive, and hence less cost-effective, than the pilot scheme. The reason is that it will become increasingly bureaucratised: excuses will be found for involving more NHS staff in its administration. It may also be vulnerable to short-cut exercises and cost-cutting measures that limit the available scope of the care plans. That's what I think, anyway, and if I'm wrong, I'm wrong.
I am reminded by all of this of what I learnt about sentencing initiatives when I was studying the criminal justice system; in fact it's a pretty widespread phenomenon. Initially, a group of ingenious, enthusiastic, and hardworking colleagues devise and run an innovative scheme that produces some really good results (note 3). The scheme is adopted as standard practice on a wider basis, overseen at higher management levels. And then somehow its effectiveness becomes less evident, enthusiasm wanes, and perhaps old practices start creeping back. Then, maybe, more novel ideas come along again and are taken up.
This is similar to a phenomenon that I recently read about in a paper that appeared two years ago in the New Yorker online (note 4). It is the tendency for scientifically demonstrated effects seemingly to become weaker (though still present) with replication over time. Firstly, the paper touches on the evidence that certain pharmaceutical treatments for mental health problems seemingly become less effective than original studies showed (I think this applies to psychological therapies as well). The paper then summarises research by Jonathan Schooler, formerly of the University of Washington, on the psychological phenomenon of 'verbal overshadowing'. First reported by Schooler in 1990, this is the paradoxical tendency for memory to be poorer for objects and sensory experiences that one has verbally described on initial presentation. Since then, with repeated replication by Schooler and other researchers, the effect size has shrunk. The phenomenon has been labelled 'cosmic habituation' by Schooler. The paper then goes on to describe other research that seems to follow the 'cosmic habituation' route, including studies of extrasensory perception, asymmetry and sexual selection in barn swallows, and treatments for certain medical complaints.
I emailed ASKE members on the New Yorker ASKEnet discussion network for their comments on the New Yorker article and was pleased to receive back the following thoughts from Niall Taylor, a veterinary surgeon in Glastonbury.
From Niall Taylor on Lehrer (2010)
This is an interesting thought piece but the answers to the questions are really all within the article itself. The author is plainly trying to make thought-provoking journalism out of all the data he has used and quoted, but a lot of it is puff. In my opinion the nub of the idea is pretty simple really.
There are a lot of things thrown together here which really shouldn't have been. On the one hand we're talking about double-blind placebo-controlled trials (DBPCTs) for a pharmaceutical, next we're on to psychology and ecology and then to ESP. There is far too much in the blend here - the author himself is showing distinct selection bias in his attempt to compile a good story.
Adam Linzmayer's amazing response to the ESP test was clearly some sort of fake - unintentional or not. ESP, and the precognition mentioned later, do not exist and basing a claim on changing results in trials of paranormal phenomena have nothing to do with the scientific process. All it means is that to start with Linzmayer was able to see the test cards and then in later trials he wasn't, not that a non-existent phenomenon changes with time for unknown reasons.
It's all very well for the article to claim, with regard to pre-cognition, 'The craziness of the hypothesis was the point: Schooler knows that precognition lacks a scientific explanation. But he wasn't testing extrasensory powers; he was testing the decline effect'. But this is just nonsense - how can you test for a decline in something which doesn't exist?
Trials in psychology and ecology are difficult to conduct in a randomised and controlled way and would naturally be more prone to experimenter bias, particularly, as the article says, when the subject is trendy and both experimenter and journal are keen to get positive results in a fashionable subject because that is what sells. As Leigh Simmons states in the article, 'The journals only wanted confirming data. It was too exciting an idea to disprove'.
Even medical trials, though supposedly more strictly controlled, are prone to bias caused by the enthusiasm of the researchers and the drive of the pharmaceutical company to succeed. Even the most rigorous protocols are unable to eliminate the lack of interest in negative or inconclusive results in the early, heady days of the launch of a 'breakthrough' drug. It even says as much in the article, but these quotes come from near the bottom, after the weird, attention-grabbing, metaphysical 'we're losing our truths' and 'cosmic habituation' stuff:
'Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results'.
'Palmer summarized the impact of selective reporting on his field: "We cannot escape the troubling conclusion that some - perhaps many - cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori beliefs often repeated."'
To my mind all this article is saying really is that the scientific protocol is flawed. It's worse in the 'soft' sciences but it still exists in DBPCTs, particularly as research in a particular field or on a 'revolutionary' theory is just beginning. It's only later, once the hype has died down (and, dare I say it, the patents have expired!) that less a priori enthusiastic workers find the real story, which inevitably is always going to be at least a little more down-beat than at the start.
I thought it was most enlightening when Jonathan Schooler said, 'I still want to know what happened to my results... I assumed that it would get easier to document my effect over time. I'd get better at doing the experiments.... So why did the opposite happen?'.
Simmons is obviously a good and rigorous scientist and one who is content to admit that his original pet theory may not be all it was when he first started to develop it. But he doesn't seem to be able to recognise this himself - to me it is obvious, if his original theory was flawed, the results would start to reflect this as he gets better at doing the experiments, not the other way around as he is expecting.
And that's it in a nutshell really - shock horror - the peer review process is less than perfect. Well, no one has ever claimed otherwise, but, let's face it, it's the best we've got!
1. Evaluation of the personal health budget pilot programme published on 30 November 2012 by the Personal Health Budgets Evaluation (PHBE) team, led by the University of Kent: here.
2. http://www.telegraph.co.uk/health/healthnews/9712708/NHS-to-pay-for-singing-lessons-and-hotel-stays.html .
3. At least in the NHS this is often how successful initiatives get off the ground when they have good support from onsite managers. Initiatives that are 'delivered' from on high by committees at Trust level often simply cause everybody more work for little in return.
4. Jonah Lehrer (2010) 'The truth wears off: Is there something wrong with the scientific method?' at:http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF.