This post is a response to a post by JP on his new blog,
SupportVaccination.org.
[Update: JP has decided to leave the vaccine discussion and take his blog down, so the link no longer works. I wish I had saved a copy of the initial post that this was written in response to, but alas, I was not forward thinking enough to save a copy.]
[Update:
John Cartan is the man! He found a cashed version for me. Here ya go:
http://tinyurl.com/7celt ]
In his post, he ask questions of parents like me, that I am eager to answer. Parents have been accused of being "scientifically illiterate" and dismissing "well designed" population studies that show that there is no link between autism and thimerosal. I thought this was a good opportunity to tell the blogesphere, at least in part, why we are not so hot on the IOM’s epidemiological studies.
Dear JP,
As one of those parents with… well ‘distain’ is a harsh word, lets call it an ‘unwillingness to rely on’ the epidemiological studies cited by the IOM, it probably falls upon the likes of me to answer the question you pose in the title of your piece.
Why the disdain for epidemiology...
Since I am the likes of me, I will take this on and answer your questions and challenges from my point of view.
I am going to break up my response to your post into a few separate posts, as there are several things to address.
I am the mother of an autistic 3 year old boy and a marriage and family therapist, with a masters degree from Johns Hopkins. The statistics courses that I took in grad school were designed to educate me on how to evaluate the research of others to see if it was something we should incorporate into the treatment of a client. This education has turned out to be useful in evaluating the epidemiological studies that have been offered up to prove that thimerosal is safe.
I have been looking at these studies with two basic questions in mind:
Can this study be applied to entire population of children in the U.S. and thereby be useful for guidance in setting vaccine policy? If so, how?
Can I use this study to make a determination as to the safety of vaccinating my sons, one who is autistic and one who is (mostly) neurotypical. If so, how?
I want to start first with what an epidemiological study is and what it can do.
Any question like this starts with case studies. Some one notices two things occurring together (smoking and lung cancer, vaccine reaction and autism) and they look at the plausibility of a relationship (smoke filling the lungs could cause damage, neurotoxic mercury in vaccines could cause brain damage).
Next they want to see if what they are observing in their setting could be happening else where and after conferring with associates and finding that others see the possibility of a link as well, they commission a large population study.
Epidemiological studies are limited in their use. They can be used to spot patterns and trends, but they can almost never be used to prove or disprove anything. They are one of the first stages of a medical inquiry and act as a divining rod to tell researchers where to start digging.
They are very vulnerable to confounders, because they are asking broad questions over huge numbers of people; so they should not be used to make definitive statements as much as to help researchers shape the next question that should be asked.
Those next questions are taken to smaller population studies (where confounders can be more easily controlled), lab studies (in vitro), and actual case studies (in vivo). The results of those studies, helps to further refine the questions being asked, which can be sent back up to large population studies, and so on, and so on until, (hopefully) the results of your epidemiological, in vitro and in vivo studies all line up like tumblers in a lock and the lock opens with one key.
This is not what we are seeing yet when we look at totality of autism/thimerosal research, as the IOM report shows. In fact the large epidemiological studies relied on for the conclusions in the report are at odds with other in vitro and in vivo research also in the report. The tumblers do not line up and therefore all of the research should be scrutinized to see what is throwing things off.
The scrutiny of the epidemiological studies has shown that they do not measure what they purport to measure, and even if they did, they cannot be applied in the way that they are being applied.
Your example of how epidemiological studies were used in the 50’s a nice example of how they can be used well and lead researchers to determine the source of a terrible illness like lung cancer, unfortunately it seems that what is going on today in this inquiry is a very different scenario.
What we know so far about autism tells us that it is not as straight forward as, smoke cigarettes get lung cancer. The thimerosal studies seem to have multiple confounders (genetic, environmental, etc) that cut into the reliability of epidemiological studies.
The two studies that are relied upon the most heavily are the
Danish Study and the
Verstraten Study. Both of these studies were designed and carried out very poorly and are being used very badly.
Here is why I treat them with such skepticism. (Quoting myself in part from an earlier post that responded to a piece by Dr. Laidler)
One of the problems that complicates the search for the causes and cures of autism is the assumption that it is a single disorder and that all cases are caused by the same thing. This is in contrast to a disease like AIDS in which all cases are brought on by the HIV virus.
What is diagnosed as ‘Autism’ is actually likely to be more than one physical syndrome. Because Autism is a DSM diagnosis and based solely on behavioral criteria, the diagnosis does not address the different sets of physical anomaly clusters that researchers are finding when they look at the biology of ‘autistic’ patients.
One research group has recently made a distinction between two types of autistic profiles. One, labeled complex autism and occurring in about 30% of autism cases, presents with smaller head size, greater chance of low IQ, lower rate of siblings with the disorder and seems to have close to a 1:1 ratio of occurrence between boys and girls.
The other, essential autism, presents with larger head size, few structural brain abnormalities and has a ratio of 7:1 boys to girls.
This is a huge confounder to any epidemiological study, even a well designed one, that looks at all cases of diagnosed autism with out differentiating between groups with different phenotypes.
In discussing research into the causes of autism (in this case the genetic cause), one of the researchers commented, “…Studies haven’t found anything because they’re looking at a big heterogeneous group — a mishmash of people with different etiologies,” Takahashi said.”
One could postulate that only the larger head size group displays autistic symptoms because of mercury poisoning, as mercury toxicity does not always present with impaired IQ; and because in vitro thimerosal toxicity studies preformed at the University of Kentucky showed that estrogen has a protective effect against, and testosterone exacerbates the damage of, ethlymercury as a neurotoxin.
In light of this new information it seems that a better question than, “is autism is caused by mercury toxicity” which is the question largely being asked in these epidemiological studies, would be, “are cases of mercury toxicity being misdiagnosed as autism”.
The answer to that question cannot be found in a large epidemiological study.
The authors of the the final IOM report on the thimerosal/autism relationship in 2004 stated themselves that the epidemiological research it was relying on to clear thimerosal as a cause of autism would not pick up on a sub-set of the population that was genetically vulnerable to mercury poisoning.
If that is the case, then how much more so to these studies fail to give us meaningful data if the children who may be mercury poisoned are members of two different subsets being confused with one another?
Since the working theory in biomedical treatment of autism is that many of the children diagnosed with autism are genetically vulnerable sub set who cannot excrete heavy metals, and since this treatment is bringing about successes in the abatement of many children's autistic symptoms, does that not throw into question the use of these large scale studies to find the correlation between thimerosal and autism as easily as they would between smoking and cancer?
What causes further problems for people who want to use these studies to show that the case is closed on thimerosal and NDDs (as the conclusion in the IOM study does) is that these studies are not well designed and are very poorly applied.
The findings in the Denmark study have come under serious criticism. When the data of study was reviewed, it was found that the sampling presented fatal flaws.
The low incidence of autism during the use of thimerosal can be attributed to the fact that the database that was used only tracked inpatient cases of autism at the time. At the point in time where thimerosal was removed, the database was expanded to include cases that were diagnosed at a large clinic outside of Copenhagen where 20% of the countries autistic patents were diagnosed. At the point in time where thimerosal was no longer used, but the cases of autism seem to have skyrocketed, the database had expanded further to include all cases of autism, inpatient and outpatient, in the country.
The sample size of the study was only 956 children. That is the number of people that their disability database had on record as being diagnosed with autism in Denmark between 1971 and 2000. 956 people in 29 years. That is 33 people a year in the entire country. [Hyperbole warning] I have that many autistic kids in my kitchen right now!
The study is further compromised by the fact that several of the coauthors were employed by the Statens Serums Institut, the countries sole vaccine manufacturer who would presumably be held liable if it was indeed found that the use of thimerosal in vaccines contributed to autism.
[Update: Kristjan has pointed out
here that the Danish health care system and liability laws are different from ours and no such legal liability exists for the Statens Serums Institut. I still wonder what might motivate these researchers to stand buy this study and not protest its poor use. More on that in the comments
here,
here and
here.]
Further, even if the study were reliable, applying it to the U.S. population as the IOM has done presents problems. Children in Denmark were administered less than half the amount of thimerosal of US children, and it was administered later and over a longer period of time.
Finally, American children, like my sons, are subject to an autism rate at least 10 times that of Denmark, so this study should not be applied to determine their risk of autism from thimerosal exposure.
It seems to me to be like doing a population study of Sickle Cell Anemia in Denmark and applying it to the population of Baltimore. Clearly U.S. children have some other intervening factors that increases the threat of NDDs, be they genetic, environmental or the thimerosal dosage and age of administration.
The problems with the Verstraten study, the only epidemiological study that the CDC has done in the U.S. to examine the possible link between thimerosal containing vaccines and neurodevelopmental disorders, are severe. It is such a badly designed and executed study that I can hardly do it justice in a blog post. I started working on a review of it about 6 weeks ago for my blog and I still have not finished.
In fact, it is currently the subject of a Senate investigation that may result in hearings on Capitol Hill.
I will try to do a quick review of my ‘distain’ for it, as I actually think the harshness of your term might apply to my attitude in this case.
Verstraten started with a nice, simple study. He had medical records of about 180,000 kids in California, a good sized sample. He broke them up into four main groups, kids who received no thimerosal, a little thimerosal, a moderate amount, and a high amount. He checked to see if the amount of thimerosal that they got, and when they got it (two months of age, six months of age, etc) corresponded to an increased risk for different types of neurodevelopmental disorders, and NDDs in general.
The study began in November of 1999 and was supposed to be finished in 4 months.
What he found was that the more thimerosal a child got, the higher their chance of an NDD, with children who got the highest doses had a huge risk of having a disorder, more than 600% in one case. He also found that the age at which it was administered determined the type of disorder that a child would develop.
Verstraten sent an email to another CDC employee, Dr. Davis, and said that no matter what he tried, could not make the relationship go away.
In December the two men, and additional researchers at the CDC, began to change the parameters of the study. When you do this, you are supposed to document all the changes and justify why they are needed to properly answer the question you are asking. They did not.
The changes they began to make to the statistical analysis of the study are now described by the CDC as “improvements”. How they could see some of these changes as improvements is unbelievable to me. Here are some of the changes:
They took the zero thimerosal group, and tucked them into the low thimerosal group. Now they only have three groups and in effect they have brought up the bottom, so the top does not seem as high.
That did not bring down the risk enough, so they decided to get rid of the no thimerosal group all together, so now you are only comparing the low, middle and high groups, bring up the bottom further.
They got rid of a catch all group of NDDs so the study no longer addressed the question, ‘does an increase thimerosal increase the risk of a neurodevelopmental disorder’. Now it only looked at each disorder separately.
Still not dampening the signal enough, the decided to go into and change some of the actual medical records of the patients they were studying. The CDC reports that these were correcting errors in patient records, but will not offer any proof of this claim, saying instead that the data that would confirm their claim was ‘lost’.
This brought the risk down, but still showed a link between thimerosal and NDDs, so they then started dropping children from the study. They used about 20 different ICD9 codes to exclude any child from the study that had almost any birth complication or whose mother had almost any pregnancy complication. This included serious problems like premature birth and birth defects, but went all the way down to moms who took an antibiotic while pregnant. In effect this removes from the study many children who are likely to go on to develop NDDs.
This is fine to do in your study if you want, but it renders it almost completely useless for application to vaccine policy. This policy covers all U.S. children, and lots and lots of those children were subject to pregnancy and birth complications.
This study now no longer applies to my children as both of my pregnancies had complications.
At this point it is safe to say that this study no longer addressed the question of whether or not an increase in the dose of thimerosal increases the chance of an NDD in American children. But the ‘improvements’ don’t stop there.
Many more people are given a chance to make suggestions, the Simpsonwood meeting was held, comments are made that the study never should have been done in the first place, and the research is further bastardized.
A stop date was put in place so that children who were initially diagnosed with something like a speech delay, were then always considered to be speech delayed, even if they went on to be diagnosed with a more serious disorder like full blown autism. This is important as a large number of kids diagnosed with autism at ages 3 and 4 are diagnosed with some milder developmental delay at 18 months or two years of age. Doctors don’t like giving the autism diagnosis early, and the state of California (where the study was done) won’t even formally evaluate children for autism, or diagnose it before the age of 3.
I live in California. My son was evaluated at 2 and found to have ‘speech delays’ and offered early start services. To all involved in his treatment, he was autistic, but he did not become officially “Autistic” according to the state and his own records until age 3. Chandler would not have been considered Autistic for the purposes of the Verstraeten study.
My favorite ‘improvement’ was to add into the study children who were only a year or so old at the cut off date of the study. No child is diagnosed with autism, or indeed much of anything, this early. They all could have gone on to be diagnosed with a neurodevelopmental disorder but in the Verstraeten study, they are all considered healthy and unaffected by thimerosal.
Even with all these shenanigans, they STILL could not completely get rid of the relationship between thimerosal and NDDs. They then employed a tactic that served to make their own findings in the study irrelevant.
They split the whole group up into two, one large one from one HMO and one smaller group from another HMO (I think it was around 16,000). The small group was now too small to be of any statistical power. They used that group to say that the results in the first group could not be replicated in the second.
Then they bought another database from an HMO in New England, which was odd because they already owned dozens of them at the cost of millions of dollars in tax payer money. The HMO had failed and was in receivership due in part to poor record keeping on out of date computers. The HMO also used their own diagnostic system that didn’t even implement ICD9 codes and the researchers used completely different parameters to study this database than they did in the first. They used this third group of only 12,000 or so patients, as yet another example of how they could not replicate the results of the first, large group, which they were now referring to as ‘the screening study’.
The 4 month study took 4 years and, in my opinion, came out looking like something akin to Frankenstein’s Monster.
This study does not offer a meaningful measure of anything and cannot be applied to any group that I can think of.
Add to this the fact Verstraten himself became an employee of Glaxo (currently being sued by parents of autistic children) half way through the study, which was not disclosed when the study was published, and it becomes easy to see why where the distain comes from.
When asked to justify all the changes and publish the data so that the study could be confirmed and replicated, the CDC repeated the claim that the original data was ‘lost’. A private contractor testified before congress that he had been ordered to destroy the data sets, “to insure patient confidentiality”. This is a violation of federal law and is what sparked the congressional investigation currently underway.
It is practice in the scientific community that if a study can not be confirmed or replicated, that it should be withdrawn. Despite parent requests for such action, the CDC stands by this study and refuses to pull it.
As a parent who is looking at this issue as hard as I can, I am upset that the IOM, who should know better, keeps calling these studies ‘well designed’ and has used them to show that the case is closed on thimerosal and autism.
Verstraeten himself says that the study is a ‘neutral study’ and does not find for or against thimerosal in the implication that it is involved with NDDs.
In grad school, in order to pass statistics we had to take studies and break them down, justifying if and how they could be used for us to make treatment decisions. The Institute of Medicine would have failed my 600 level stats courses.
ADDENDUM:
People have asked for citations. What hasn't been referenced above can be found in David Kirby's book, Evidence Of Harm as this post just attempts to squish his big fat book into a blog post.
Additional addendum:
Julie Gerberding was asked by Congress to defend Verstraeten and had to reply that it was a useless study. She did so in secret, but the document was leaked by a congressional staffer. Even though everyone knows it is garbage, Pediatrics has not retracted it, and some still claim it clears the vaccine/autism theory. However CDC removed it from their list of research that refutes the theory in by the end of 2008 when
news of the Congressional/CDC exchange went public.
