As concerned parents or adults affected by ADHD and related disorders, we see a lot of headlines designed to grab our attention. Many of them sound like they are reporting new scientific findings, and some of them are… but others are peddling their wares and hoping to influence consumers or impress them with false authority. Every responsible consumer should know how to critically analyze an article or study, especially when the it concerns their health and money.
Many of these press releases and articles do not link to any studies at all, but rather reference “research” in a nebulous way. Others may use buzz words and scare tactics or rely on implied authority. Others may appear like legitimate reporting but may actually be selling something. The ‘Fish Oil’ headline above, for example, looks like a press release, but scroll down in the “article,” and you’d see that it’s an ad for a supplement and that it doesn’t even link to any research at all!
How to Spot a Dubious Source
When evaluating the credibility of a source, first examine the author’s credentials and institution affiliations. Some are well-respected (major universities and government institutions), but others can appear well-respected but are agenda-driven (the Discovery Institute).
When evaluating an author making any big claims, you should examine not just their degree, but also if they have any credentials from the field or any peer-reviewed published research.
An interesting example of an author with a lot of perceived clout but with dubious credibility is Dr. Perlmutter, author of Grain Brain. He has been something of a sensation for the last couple of years, making headlines with his recommendations for a gluten-free diet, even claiming that ADHD and Autism may be prevented with the use of probiotics and treated with a grain elimination diet.
“[Dr. Perlmutter] contributed extensively to the world medical literature,” reads his website, “with publications appearing in The Journal of Neurosurgery, The Southern Medical Journal, Journal of Applied Nutrition, and Archives of Neurology.” Earlier bios also list the Journal of the American Medical Association.
Yet a closer look at his publications reveals that Perlmutter hasn’t actually conducted much research. In the scientific community, full-length peer-reviewed articles, especially widely cited ones, are the gold-standard of significant research. But his contribution to JAMA — an extremely prestigious medical journal — is actually just a letter to the editor. The Southern Medical Journal? A case report and a clinical brief, both co-authored with his father when the younger Perlmutter was still a medical student. Archives of Neurology? Another case report. (Case reports and clinical briefs are short discussions around 1-4 pages long.) As for the prestigious-sounding Journal of Applied Nutrition, you’ll be hard-pressed to find a legitimate nutrition scientist who’s even heard of it.” -via Alan Leninovitz for The Science of Us
When you consider the above, it may seem wise to take Dr. Perlmutter’s recommendations with a grain of salt, no?
A little digging may find that an author may have an interest in promoting a particular viewpoint or even a product. Dr. Perlmutter happens to direct readers to his $8,500 brain detoxification retreat, and among many other products, he sells $160 “Senior Empowerment Packs,” and he also used to endorse his own expensive hyperbaric oxygen chamber treatments!
An important point to remember: if an author has affiliations with a particular organization with an agenda, then what they have to say about matters relating to that topic must be understood to be influenced by their interest in the matter.
Questions to Ask:
- Does this information come from an unbiased source?
- Is the information presented factual and relevant?
- Are the claims backed up by statistically significant studies and cited?
If the article only presents one side on the matter and ignores or completely discounts all other viewpoints, it’s biased. It may be a valid opinion piece, but it is not unbiased and it is not research.
A couple more examples to consider:
There has been a lot of buzz on Twitter lately about Neurocore, a company claiming to treat ADHD and Autism with neurofeedback. The site even makes a claim to raise a person’s IQ “an average of 12 points” using their technique! Their Chief Science Officer is Timothy Royer, the “former division chief of pediatric psychology” at Helen DeVos Children’s Hospital in Grand Rapids, MI. That would him a kids’ therapist, but quite an enterprising and ambitious one to go into business peddling expensive treatments and unnecessary scans to concerned parents who want to do right by their children.
“Timothy G. Royer, Psy.D. is an expert in the use of biofeedback and autonomic nervous system (ANS) regulation to treat ADHD, sleep loss, anxiety and other stress disorders in children and adults.” – via the Neurocore site
“Using data from quantitative electroencephalography (qEEG), along with other physiological measures, Neurocore gains a deep understanding of why a person’s health and well-being may be compromised. Neurocore has provided diagnostic qEEG to over 10,000 children and adults.”
“Once the diagnosis is completed, Neurocore uses proven neurofeedback therapy to train the brain to operate more efficiently. Nearly 75% of individuals who have completed the neurofeedback program have eliminated or greatly reduced their medications for ADHD and experience improved performance and behavior. Adults report better sleep, renewed focus, and significant relief from stress.” – via Neurocore
Does Timothy Royer (who calls himself Dr. Royer on Twitter) pass the credibility test?
Well, his PsyD degree is in psychology, not medicine, and he has no published research to be found. His site does not provide links to research studies that would indicate how he might substantiate his claims about raising the IQ with his technique or to back up his claims of so many patients who no longer require medication after his treatment. Finally, he most definitely has a vested interest in selling his product: a very expensive treatment program which may not be covered by insurance, leading one to question the efficacy of the treatment.
While Royer’s claims are interesting and appealing to anyone who depends on medication to function normally, without conducting any research studies or publishing his findings for peer review, his claims remain unsubstantiated and anecdotal at best.
Joseph Mercola of mercola.com is another prime example of a dubious source. His articles often cite news articles or quotes from other “experts” with an agenda instead of peer-reviewed research and rely on circular reasoning and the logical fallacy of implied authority because he’s a “doctor” (so he must be correct). In ADHD Experts Re-evaluate Their Zeal for Drugs, he says:
“These are hard-core, “class 2″ narcotics, regulated by the Drug Enforcement Agency as a controlled substance because they can lead to dependence. The majority of kids diagnosed with ADHD will be prescribed these potentially dangerous drugs, the most common being Ritalin. By definition, Ritalin stimulates your central nervous system and may certainly interfere with the delicate and complex workings of your brain and personality.”
The problem with this passage is that “by definition,” many useful drugs are central nervous system stimulants. This does not make them harmful when used as prescribed. And by using the term “hard-core,” he attempts to confuse the issue with the fallacy of relevance. Yes, stimulants like methylphenidate are classified as Class 2 and carry a risk of dependence (like many other medications), but these medications are not normally abused by the children and adults to whom they are prescribed.
Joseph Mercola has been recognized as a quack doctor for awhile. He has been criticized by regulatory, medical, and scientific communities. A 2006 BusinessWeek editorial called his marketing practices as “relying on slick promotion, clever use of information, and scare tactics.” The U.S. Food and Drug Administration has even warned Mercola and his company to stop making illegal claims regarding his products’ ability to detect, prevent, and treat disease.
“Many of Mercola’s articles make unsubstantiated claims and clash with those of leading medical and public health organizations. For example, he opposes immunization, fluoridation, mammography, and the routine administration of vitamin K shots to the newborn; claims that amalgam fillings are toxic; and makes many unsubstantiated recommendations for dietary supplements. Mercola’s reach has been greatly boosted by repeated promotion on the “Dr. Oz Show.”” – via QuackWatch
How to Recognize a Credible Source
If a source is credible, the headline will be reasonable, rather than sensational, and the information will be presented factually, acknowledging the weaknesses in the study or the opposing viewpoint. The information should be well organized and with logical arguments, and claims will be backed up with research from other credible sources from research conducted in the last few years. Consider the source of the article, but also the source of the citations (The New York Times or similar sites/papers are not research journals and cannot be cited as references).
Finally, even if the source appears to come from a journal, not all journals are reliable. Some are “standalone journals” that allow submission from almost anyone or have less than rigorous review methods. When checking the validity of a published journal article, see if it’s a standalone journal here.
Let’s consider the following, which poses a common question:
This was found in the prestigious Journal of the American Academy of Child and Adolescent Psychiatry. JAACAP is the #1 journal in the category of pediatrics, as reported in the 2013 Journal Citation Reports (the recognized authority for evaluating journals). JAACAP also ranks 11th of 135 psychiatry journals.
The lead researcher/author of cited is Dr. Peter S. Jensen, a psychiatrist (a Medical Doctor specializing in psychiatric illnesses) and the Associate Director for Child and Adolescent Research, at the National Institute for Mental Health. He has 233 publications to date. See the full list here.
His professional highlights include
- Member, Scientific Council, NAMI, 2007-present
- President, The REACH Institute (Resource for Advancing Children’s Health), 2007-present
- Irving Philips Prevention Award, American Academy of Child & Adolescent Psychiatry, 2011
- Alumni Inductee, Alpha Omega Alpha, 2008
- Editorial Board Memberships: Journal of Child & Adolescent Psychopharmacology, 1990-present; Development and Psychopathology, 1993-present; Journal of Attention Disorders, 1995-present; Biologic Psychiatry, 1997-present; J. Abnormal Child Psychology, 1998- present; World Journal of Clinical Pediatrics, 2011-present; World Journal of Translational Medicine, 2011-present
Among the many others who participated in the research are Dr. Dulcan, the Head of the Division of Child and Adolescent Psychiatry, Northwestern University School of Medicine; Drs. Hoven and Bird with the Department of Child Psychiatry, New York State Psychiatric Institute; and Dr. Bauermeister with the Department of Psychiatry, University of Puerto Rico.
In this study, the authors examined epidemiological survey data from 1,285 children and their parents across 4 U.S. communities. They reported objective and verifiable facts gathered from their study, and shared their methods, results, and conclusions:
“Findings indicated that 5.1% of children met full DSM-III-R ADHD criteria across the pooled sample. Only 12.5% of children meeting ADHD criteria had been treated with stimulants during the previous 12 months. Some children who had been prescribed stimulants did not meet full ADHD diagnostic criteria, but these children manifested high levels of ADHD symptoms, suggesting that the medication had been appropriately prescribed. Children with ADHD were generally more likely to receive mental health counseling and/or school-based interventions than medication.” – from JAACAP
“On the basis of these data it cannot be concluded that substantial “overtreatment” with stimulants is occurring across communities in general.”
This passes the quack test. The source is highly credible, the affiliated instituation and publishing journal are reputable and trustworthy, the findings are not sensationalized, the data are objective, and the data are from a large enough sample size to be worth considering (though a larger survey will always provide even better results).
Not all research studies are created equal. Depending on the sample size and the methods used, the significance of the results can vary greatly.
I plan to get into that later this summer with my next post in this series: How to Analyze a Research Study
Did you find this post helpful? Please SHARE it!