Last year, we updated our Cochrane review on mammography screening with more mortality data and sent it to Cochrane in August 2023. The many comments we received on the very minor, fourth update of a well-known and highly cited Cochrane review that has been around for 23 years1 illustrates that Cochrane has become a highly bureaucratic and ineffective organisation, which is one of the main reasons that all the UK based Cochrane groups lost their governmental funding at the end of March 2023.2
Although we had not included any new trials, it took five months before we got a reply. In January 2024, we received a document to which 11 people had provided comments, 8 from Cochrane and 3 peer reviewers. There were 91 separate, numbered comments over 21 pages (7,290 words, or the size of two normal scientific articles).
We replied to the 91 comments and sent a revision of the review to Cochrane two months later. The document was now 12,559 words. We had concerns about some of the comments, and the most important ones were these:
1 Cochrane comment: “There is no presentation of the result of data on all the eligible trials as a group.”
Our reply: “We did not present a summary estimate for the low- and high risk of bias trials combined. This is in accordance with accepted standards when differences in risk of bias between trials can explain the heterogeneity. Here, there were substantial differences between the most (> 40% reduction) and least (no benefit) trials, and confidence intervals did not overlap. In such cases, one should trust the low risk of bias trials and a summary estimate that includes both low and high risk of bias trials would provide a misleading effect estimate.”
2 Cochrane comment: “Measuring overdiagnosis is controversial, lacking agreement on how to estimate this theoretical construct … Please only include outcomes which have been clearly defined and transparently measured in the results sections of the discussion and summary sections of the review.”
Our reply: “Our analysis of the increase in breast cancer diagnoses and the interpretation that this constitutes overdiagnosis is the same as in the review from the UK Independent Panel (Marmot et al.)3 … Overdiagnosis in breast screening has taken a long time to become commonly accepted as the most important harm. Any systematic review that does not aim to quantify overdiagnosis would be deficient and would fail to allow an informed choice.”
3 Cochrane comment: “In the section ‘Authors conclusions > implications for practice’, authors should use language that accurately reflects the certainty of results. For example, statements such as ‘The most reliable trials did not support that breast screening reduce breast cancer mortality for any age group.’ should be rewritten to read ‘The most reliable trials indicated low certainty evidence that breast screening may make little to no difference to breast cancer mortality for any age group’. (see table 15.6.b for more guidance https://training.cochrane.org/handbook/current/chapter-15#section-15-6-4).
Our reply: “We disagree that the recommended wording is helpful to readers and the GRADE language is likely quite unclear to others than those working with GRADE on a regular basis. It is quite far from the plain language we generally strive for in Cochrane reviews. Essentially, the meaning of the two texts is the same: the burden of proof for this intervention has not been lifted.”
4 Cochrane comment: “In the section ‘Abstract > main results’, authors must ensure that they use wording that reflects the full range of possibilities indicated by the results. For example, as stated in section 15.6.4 in the Cochrane Handbook, ‘If the confidence interval for the estimate of the difference in the effects of the interventions overlaps with no effect, the analysis is compatible with both a true beneficial effect and a true harmful effect. If one of the possibilities is mentioned in the conclusion, the other possibility should be mentioned as well’. Therefore, statements such as ‘did not show a benefit in terms of a reduction in breast cancer mortality’ should be amended to read ‘may show little or no difference in terms of a reduction in breast cancer mortality’. (See https://training.cochrane.org/handbook/current/chapter-15#section-15-6-4). Similar edits should be made throughout this section.”
Our reply: “We find the recommended wording unclear and unnecessarily confusing. The question is if the trials can lift the burden of proof that an intervention is beneficial and in this case they cannot … Furthermore, it is subjective whether a difference is ‘little’ or not, which we therefore prefer to avoid.”
5 Cochrane comment: “In the Plain Language Summary, authors are advised to ensure they use language that accurately reflects the certainty of the evidence. For example, the statement ‘The most reliable studies did not show that breast screening with mammography reduces your risk of dying from breast cancer’ should be amended to read ‘The most reliable studies show uncertain evidence that breast screening with mammography may make little to no difference on the risk of dying from breast cancer.’”
Our reply: “We find the suggested sentence very complex and not suitable for a plain language summary intended for others than those who work with GRADE on a daily basis. Our review asks the question if the burden of proof has been lifted for breast screening and we find that this is not the case. In our opinion, our sentence in the Plain Language Summary clearly expresses this.”
We felt we had responded adequately to the comments and expected the update to be approved quickly.
After 2.5 months, we became a little impatient and inquired about the progress with our review. We noted that, a couple of days earlier, the Canadian Task Force on Preventive Health Care had published new draft recommendations on mammography screening, which would obviously cause controversy and protests, and an update of our review was therefore highly timely.
Liz Bickerdike, Senior Managing Editor at Cochrane, apologised for the delays: “Unfortunately, one of the sign-off editors is unavailable which has led to some delays in assessing your revised draft review. I appreciate delays are frustrating. Please be assured that we are working to address this as quickly as we can. I expect to be in touch regarding an editorial decision by the end of the month.”
We replied that, “given the rapidly evolving situation in this area where two major guideline groups (the U.S. Preventive Services Task Force and the Canadian Task Force on Preventive Health Care) have sent out conflicting recommendations within the past month, we believe it is very important for an informed debate that our update is made available to the public and to decision makers. As it seems unlikely that our updated Cochrane review will see publication this side of summer vacations in the northern hemisphere, we will upload it to medRxiv.org …”
Bickerdike replied that, “Cochrane does not currently have a specific preprint policy, and as such, we advise authors not to upload preprints of unpublished reviews to online preprint servers. Should you choose to upload the draft please note that it cannot represent an affiliation to Cochrane and cannot be described as a Cochrane Review.”
This raises three interesting questions. First, if an editor is unavailable in a big, professional organisation, shouldn’t there then be another editor who can sign off on a very important review? Second, as far as I know, all prestigious medical journals accept publication of preprints, so why not Cochrane, which is supposed to provide up-to-date information of importance for decision making in healthcare? Third, why the obsession with the “Cochrane brand?” As my co-author is employed at Cochrane Denmark, he cannot possibly avoid representing his affiliation to Cochrane in a preprint. And we cannot avoid that our preprint is of a Cochrane review, which everyone can see it is.
In the public interest, which Cochrane should be concerned about but isn’t, I uploaded our draft Cochrane review as a preprint.4
It seems that Cochrane has not learnt from past mistakes. When I first published the review in 2001, the Cochrane Breast Cancer Group would not allow us to publish the harms of screening even though they were included in the protocol for the review, the group had approved and published.5 We therefore had to publish our full review in The Lancet to ensure the harms became known.6,7 It took five years, with complaints to the Cochrane Steering Group, before the harms data became included in the Cochrane review.8 FIVE YEARS!
In 2001, the editors of the Cochrane Breast Cancer Group insisted that “the review in The Cochrane Library is stronger” whereas I noted that the version we published in The Lancet “lives up to Cochrane standards better.”9 Our Lancet review was much more of a Cochrane review than the Cochrane review. It was the same, but also included the harms of screening, and it did not lump unreliable studies with reliable studies, which the Cochrane editors had forced us to do, even though it is misleading.
Lancet’s editor, Richard Horton, wrote that, “if a difference remains, let the scientists doing the review publish what they wish to say – it is, after all, their work. The editors can present their own view as a supplementary discussion or comment. That way, the debate proceeds properly, each side is given its voice, accusations of censorship are avoided, and the public sees science as a truly collaborative process, in which differences of opinion are not only respected, but also welcomed.”9
I wish medical journals would heed this advice, which they, including Lancet itself, unfortunately often don’t do.
When I showed the draft of this article to a colleague who is very familiar with Cochrane, she responded: “Seems like Cochrane is up to its old tricks … obfuscate and delay.”
Time will show how the editors will handle our important objections to some of their proposals, and how long it will take before our updated review appears in The Cochrane Library. I find it particularly odd that Cochrane staff asked us to communicate that when the confidence interval includes 1, the intervention could be both beneficial and harmful, but then advised us to write: “may show little or no difference in terms of a reduction in breast cancer mortality.” This wording excludes the possibility that screening could increase breast cancer mortality, and “little” is subjective and should not be used. How little should little be in order to be little, and how far can the confidence interval be from 1 before little becomes not little?
References
1 Gøtzsche PC, Jørgensen KJ. Screening for breast cancer with mammography. Cochrane Database Syst Rev 2013;6:CD001877.
2 Gøtzsche PC. Decline and fall of the Cochrane empire. Copenhagen: Institute for Scientific Freedom 2022. Freely available.
3 Independent UK Panel on Breast Cancer Screening. The benefits and harms of breast cancer screening: an independent review. Lancet 2012;380:1778–86.
4 Gøtzsche PC. Screening for breast cancer with mammography: updated review. Institute for Scientific Freedom 2014;June 7.
5 Horton R. Screening mammography: an overview revisited. Lancet 2001;358:1284–5. Free access.
6 Olsen O, Gøtzsche PC. Cochrane review on screening for breast cancer with mammography. Lancet 2001;358:1340-2.
7 Olsen O, Gøtzsche PC. Systematic review of screening for breast cancer with mammography. Lancet 2001;Oct 20.
8 Gøtzsche PC. Mammography screening: truth, lies and controversy. London: Radcliffe Publishing; 2012.
9 Horton R. Screening mammography: setting the record straight. Lancet 2002;359: 441-2.