‘Fatally flawed’ review puts Aus research council’s methods in doubt

The National Health and Medical Research Council (NHMRC) has a statutory responsibility to provide advice on Australian public health matters, and is in the process of examining a number of alternative therapy claims.

To this end, it has conducted a review of published literature addressing the effectiveness of homeopathy with the aim to develop guidance to assist consumers in making informed decisions about their healthcare.

Homeopathy is defined by the World Health Organisation as the practice of using highly diluted preparations with the aim of treating illness. This 200-year-old form of alternative therapy is based on the belief that even though it is highly diluted, the molecules retain a “memory” of the original substance.

Laundry list of criticism​

The NHMRC’s review concluded that “the assessment of the evidence from the research in humans does not show that homeopathy is effective for treating the range of health conditions considered​”.

However, documents released under Australia’s freedom of information laws show widespread criticism by a number of independent experts, who have called the NHMRC’s research into question, and expressed strong concerns over the methodology it council employed in the review.

The documents, seen by FoodNavigator-Asia​, show that the NHMRC excluded randomised controlled trials (RTC) from the review, ignored non-English studies, didn’t call on the opinions of experts and conducted selective research through a narrow choice of databases. 

In one redacted email response to the homeopathy working committee in January 2013, a member of the NHMRC’s complementary and alternative medicines team explained how the group had decided to ignore the data contained in one academic database because “it would incur additional costs to NHMRC”, despite conceding there was a possibility that other databases would add to quality of the review.

Out of a possible 1,367 studies available for examination, the council chose only to look at 61.

In another document, one correspondent whose name was redacted queried how systemic  the NHMRC’s review of high-quality RTCs had been.

“For more than a decade, high-quality RTCs have been regarded as Level 1 evidence… The NHMRC’s decision not to adhere to a search of all Level 1 evidence as per international standards needs to be carefully explained and justified​,” the person wrote. 

“The dismissal of positive SRs [systematic reviews] compounded with the lack of an independent systematic review of high-quality RCTs leaves me uncertain of the definitive nature of the report’s conclusions​.”

The correspondent added that he or she was concerned that no homeopathic expert was appointed by the review panel, as would happen in other disciplines.

Question mark over future reviews​

Another commenter called on the NHMRC to explain why it made the decision to include only English publications within a specific period of 16 years instead of “the totality of relevant literature​”.

Carl Gibson of Complementary Medicines Australia was scathing of the “fatally flawed​” paper, saying: “A number of independent experts in the sector have expressed strong concerns with [its] methodology​.

“The NHMRC failed to appoint a homeopathic expert to the review panel, left out randomised controlled trials, excluded all studies not published in English, and limited the choice of databases searched, which basically meant that the balance of complementary medicines specific content was omitted​.

“No valid conclusions can be drawn from this review, except that the NHMRC has failed to uphold its own standards of ethics and quality research in this instance.​”

As the first of NHMRC investigations into evidence on complementary medicines, this has left Gibson concerned about other reviews in the series.

“CMA questions how future reviews will be underpinned by best practice principles and conducted in a transparent manner​,” he said. 

The NHMRC has been approached for comment.