Secret Science

US EPA has revised its rules regarding the use of “secret science” in its rulemaking processes. The revised rule now gives preference to scientific studies for which underlying data and models are available in sufficient detail to permit independent validation. This revision softens the position regarding reliance on “secret science”, largely in response to concerns about the security of study data which would permit identification of study participants. However, in almost all cases, the identity of study subjects could be protected while still providing sufficient study detail to permit validation of study methods and conclusions.

This EPA rulemaking draws attention to the broader issue of providing access to study data and models on scientific studies which do not involve human subjects. There is an acknowledged reproducibility crisis in science not limited to EPA epidemiological research. EPA has begun to address the issue, but it is long past time for other research funding agencies to address the issue as well.

Government funding agencies are in a strong position to address the issue. Requests for Proposals (RFPs) can stipulate that all study methods, data, analytical approaches and models be publicly disclosed at the conclusion of the specific project. Compliance could be assured by conditioning final payment on acceptable disclosure as stipulated in the RFP.

Scientific journals are also in a strong position. Journals can refuse to submit research papers for peer review unless acceptable disclosure has occurred. Journals can also refuse to publish research papers if peer reviewers are not satisfied with the authors’ disclosures. How is meaningful peer review possible if the reviewers do not have access to study methods, data, analytical approaches and models? Journals can also increase confidence in the peer review process by including a skeptical peer among the reviewers and avoiding “pal review”, especially the practice of allowing authors to select the peers they wish to review their work.

Privately funded research could also be subjected to rigorous peer review through the use of Non-Disclosure Agreements (NDAs), though this is more problematic. However, corporations planning major investments to commercialize the results of their research might well encourage critical peer review prior to investment decisions. Corporations also face potential legal liability if they produce ineffective or dangerous products resulting from their research.

The broader scientific community should insist on more thorough disclosure and more rigorous peer review to reverse the current irreproducibility crisis and avoid major issues in the future.

The climate science community should be particularly sensitive to these issues, as should the government agencies funding climate science research. The refusal of noted climate scientists to provide disclosure regarding controversial results, such as the infamous “hockey stick”, and the stated willingness of other climate scientists to “lose data” rather than sharing the data with skeptical scientists are clear indications of potential problems in the climate science community. Efforts to prevent the publication of research results which do not support the consensus narrative, the exclusion of such research results from consideration in the IPCC Assessment Reports and attempts to delegitimize skeptical scientists are further indications of problems in climate science.