The Comroe and Dripps Report on the Value of Basic
Research
Stephen R. Kaufman, M.D.
Proponents of basic research maintain that "knowledge for knowledge's sake" often proves valuable in unexpected ways, and that basic research is a vital component of our national research program. It has not been demonstrated, however, that basic research is either a necessary or cost-effective component of clinical advancement. In 1976, Cornroe and Dripps published a study that purportedly demonstrated a strong relationship between basic research and medical progress.(1,2) This study has been widely cited as "proof" of basic research's utility, often by scientists encouraging government support of basic research. Comroe and Dripps' methodology, however, was seriously flawed.
Part 1: Critique of Comroe and Dripps
Comroe and Dripps eschewed anecdotes as a way of assessing research's value, stating that subjective choice of anecdotes could be used to endorse any type of research. Their own method, they claimed, had employed more objective criteria to determine basic research's impact on medical progress. First, they had determined the top ten clinical advances in cardiovascular and pulmonary medicine and identified the "essential bodies of knowledge" they considered necessary prerequisites to achieving these ten advances. Next, they had decided which "key" reports had been most instrumental in providing the essential bodies of knowledge. Finally, Comroe and Dripps read each of these papers to determine whether or not the research had been clinically oriented and whether or not the research had been basic research.
Selection of the Top Ten Clinical Advances
Comroe and Dripps sent a list of major advances in cardiovascular and pulmonary medicine -- advances selected by 40 physicians -- to cardiovascular and pulmonary specialists. The specialists then selected which advances they would rate among the top ten. Thus, the final determination of the ten most important clinical advances relied on subjective opinion. Certainly, this subjectivity reduced the study's validity. Comroe and Dripps could easily have used a more objective method -- for example, analysis of therapies that randomized, controlled studies have shown to be efficacious.
Identification of Essential Bodies of Knowledge
With the assistance of 161 consultants, Comroe and Dripps attempted to identify the "essential bodies of knowledge" that necessarily preceded the ten clinical advances. Approximately half of the consultants were clinicians; the other half included basic medical scientists, engineers, science administrators, and science writers. With the help of the consultants' suggestions, Comroe and Dripps then arrived at 137 "essential bodies of knowledge." As with determination of the top ten advances, personal beliefs and biases were in no way prevented from affecting the outcome, They are likely, in fact, to have played a substantial role.
Identification of the Key Articles
Comroe and Dripps examined 6,000 published articles; of these, they selected 3,400 "particularly important to the development of the 137 essential bodies of knowledge."(2) Comroe and Dripps did not state how they obtained this list of 6,000 articles or their criteria for eliminating 43% of them from further consideration. While everyone has opinions and potential biases, Comroe and Dripps also had political concerns likely to influence their subjective decisions. Basic scientists themselves, they were disturbed by the Johnson Administration's critical view of basic research. Prior to their study, Comroe and Dripps noted, "The President's remarks have been summarized as 'research is fine, but results are better' and 'we know all we need to know; now all we must do is to apply what we already know.'"(3) Apparently, Comroe and Dripps were not disinterested observers. Indeed, as early as 1969, Comroe was convinced that a detailed review of medical history would prove basic research's value. He wrote, "An 'Operation Hindsight' in biology and medicine will show that almost every major advance in health depended on a prior basic scientific discovery."(4) Given the strong assumptions with which Comroe and Dripps undertook their study, its heavy reliance on subjective analysis seems particularly inappropriate.
Key Articles
With "the advice of consultants," Comroe and Dripps then selected 267 key articles. This report states:
Because these key articles formed the basis of our analysis, we devoted considerable thought to their selection. We realized that bias in selecting them could invalidate our study and that their careful review by consultants was essential.(1)
"Review by consultants" hardly ensures freedom from bias. Like Comroe and Dripps' own choices, those of the consultants were personal and subjective. The final decisions, too, were purely Comroe and Dripps', reflecting their own interpretation of medical history. This methodology completely begs the question that the review addressed. Comroe and Dripps sought to demonstrate which kinds of research were responsible for medical advances. Their choice of articles, however, necessarily reflected their beliefs about the relative importance of different contributions. The process, therefore, involved a fundamental circularity. Comroe and Dripps could have used their approach to endorse any research program, depending on which articles they chose. Under the circumstances, it was virtually impossible to objectively identify the key articles.
Making matters even worse, Comroe and Dripps gave "key article" this broad definition:
It reported new data, new ways of looking at old data, a new concept or hypothesis, a new method, new drug, new apparatus, or a new technique that either was essential for full development of one or more of the clinical advances.. . or greatly accelerated it.(2)
Which criteria were more important? Most likely, many of 3,400 articles satisfied at least one of them. Given Comroe and Dripps' commitment to basic research, they should have minimized their own contribution to the selection process. For example, they could have sent related groups of papers to medical historians and asked them to indicate which they consider most important.
Comroe and Dripps acknowledged the potential for bias but maintained that an additional test validated their study. They sent 42 lists of papers to other reviewers. (Each list pertained to a particular "essential body of knowledge.") The reviewers then selected "key articles." Finally, Comroe and Dripps determined whether or not the selected articles were "clinically oriented." They found, "Although there was not complete agreement on the selection of individual key articles, there was almost exact agreement on the type of articles selected. Thus the percentage of key articles reporting research that was not clinically oriented was almost identical in their selections and in ours."
This portion of the study, too, is riddled with problems. First, there are a number of ways in which bias may have invalidated the results. The background of the reviewers was not provided; they may have shared Comroe and Dripps' biases. Also, they may have overestimated the value of their own research or that of scientists working in their own field. Similarly, reviewers may have chosen articles authored by "big names" widely publicized as having made major contributions. Scientists' prominence is often increased by eloquence or political maneuvering, quite apart from the relative importance of their discoveries. Interestingly, two papers listed as "key articles" were co-authored by Comroe,(5,6) but the report did not specify whether Comroe and Dripps themselves or other reviewers chose these articles.
Second, the list of key articles selected by the reviewers differed substantially from the list of Comroe and Dripps. Only 128 of the 494 articles chosen by the reviewers were also selected by Comroe and Dripps. Thus, 74% of the reviewers' choices were not chosen by Comroe and Dripps, and 52% of the key articles chosen by Comroe and Dripps were not chosen by the reviewers, Comroe and Dripps explained, "(i) the reviewers on the average selected 8.4 key articles per table and we selected on the average only 6.7 for these 42 tables; and (ii) we sent some tables to more than one reviewer." These differences in selection process can account for a small amount of the discrepancy, but the large difference in choices reflects considerable subjectivity in choosing "key articles."
Third, while other reviewers designated almost exactly the same number of clinically oriented choices, Comroe and Dripps did not state how well or poorly the reviewers' percentage of "basic science" articles accorded with their own percentage. Such an analysis would have been easy. Why weren't these data reported? Did the reviewers' choices differ markedly from Comroe and Dripps'?
Finally, when they reported their findings in Science in April 1976, Comroe and Dripps noted that they had reviewed "over 4,000" papers. By the time they published their final report in January 1977, the number of articles considered had grown to "about 6,000." Why weren't these extra 2,000 articles part of the initial analysis? Once additional important articles were found, did they redo all the lists of key articles? Did they merely add more "key articles" from these 2,000 articles without weighing these choices against the other 4,000 articles? Comroe and Dripps failed to clarify these points.
Determination of Clinically Oriented Articles
Comroe and Dripps determined whether or not an article was clinically oriented on the basis of the article's own wording. They discounted other historical data, including any autobiographical statements by the authors. Were their determinations accurate?
Unfortunately, Comroe and Dripps did not state which papers they categorized as clinically oriented, nor which they categorized as basic research. Later investigators cannot check their data; consequently, there is no way to assess the soundness of Comroe and Dripps' judgments. The final report lists all 6,000 papers Comroe and Dripps reviewed, with an asterix next to those that were considered key articles. Furthermore, they did not indicate which key articles they themselves chose and which were chosen by their reviewers. Again, they could have easily provided this information. Its omission prevents independent validation of their findings.
Yet another problem arises from Comroe and Dripps' reliance on a paper's own wording in their determination of whether or not that paper was clinically oriented. While many of the authors may have had clinically oriented goals, they may not have wished to speculate, when writing up their findings, on possible clinical applications. The degree to which an author speculates depends on many factors, including personal style, anticipated audience, and the journal publishing the report. Black has noted:
[Comroe and Dripps'] discriminant between 'clinically' and 'non-clinically' oriented articles seems to me to bear much too heavily on the subjectivity of the authors of the key papers, in stating their aims. A basic scientist, writing for a physiological journal, might well fail to declare an interest in some disease state which had sparked off his research; likewise, a clinician might regard his interest in disease as too obvious to require a formal statement. On the other hand, a scientist whose real and proper concern was to add to basic knowledge might well adorn his paper with a claim to clinical relevance.(7)
Basic Research's Value
Comroe and Dripps' stated goal was to determine whether or not basic research is important in medical advances. Their definition of "basic research," however, was unusually broad: "We classify research as basic when the investigator, in addition to observing, describing and measuring, attempts to determine the mechanisms responsible for the observed effects."(2) Based on this definition, they concluded that 61.5% of the key articles reported basic research. This approach creates several problems. First, the "basic research" category included any clinical study that involved some attempt to understand the mechanisms of the disease under consideration. Classifying clinical studies as "basic research" can easily mislead, since scientists generally consider "basic research" to be laboratory-based. Again, there is no way of judging the soundness of Comroe and Dripps" determinations: They did not specify which key articles they classified as basic research.
Second, defining "basic research" in terms of mechanisms violates standard practice. Smith has noted, "Most definitions are centered on the research being curiosity driven and not having any clearly foreseeable usefulness."(8)
Third, earlier key articles may have reported basic physiological principles, while more recent key articles may have developed out of applied research. Unfortunately, this possibility cannot be tested with Comroe and Dripps' data because they did not indicate which papers represent "basic" research and which do not.
Finally, one might argue that research on the mechanism of disease is actually applied research. Unlike "knowledge for knowledge's sake," investigation of the mechanism of disease is clearly intended to bear on management of the disease. Black has observed:
. . . 61.7% of the 529 'key articles' were 'basic'. Again, I would be inclined, on their definition, to substitute the word 'good' for 'basic', which may diminish somewhat any surprise at their conclusion. A research-worker who is interested not only in 'what?', but 'why?', or at least 'how?', seems to me the man who is most likely to produce useful knowledge.(7)
Comroe and Dripps' Conclusions
Comroe and Dripps concluded that their study justifies enthusiastic support of basic research. They wrote, "Basic research, as we have defined it., pays off in terms of key discoveries almost twice as handsomely as other types of research and development combined."(1) Without knowing what percentage of research funds was devoted to basic research, Comroe and Dripps have no basis for such a claim. Perhaps basic research received a disproportionately large percentage of support. Indeed, this is likely, given Comroe and Dripps' broad defmition of "basic research."
Unanswered Questions
Comroe and Dripps' study required considerable effort. Nevertheless, the results are of minimal value. Even though the report was several hundred pages long, the methodology was vague and vital data are missing.
It is impossible to check Comroe and Dripps' specific analyses to verify their validity. The study's omissions raise doubts as to fundamental scientific integrity. Such disturbing questions as these remain: Why did Comroe and Dripps not identify which articles were considered clinically oriented and which were not? Why didn't they indicate which reviewer assessed each group of articles? Were these reviewers, who purportedly served as a control, basic scientists who shared Comroe and Dripps' political concerns? Why didn't Comroe and Dripps indicate the percentage of basic science articles chosen by these reviewers?
Part 2: Use of Comroe and Dripps' Report to Influence Public Policy
Despite the obvious shortcomings of Comroe and Dripps' report, it has been widely cited. The Science Citation Index lists 96 references to the study from 1976 to 1990. A review of 38 of these papers, which appeared in leading journals such as British Medical Journal, Lancet, Nature, and New England Journal of Medicine, revealed that the study was often used to encourage government support of basic research.
The weaknesses of Comroe and Dripps' study should have been obvious to any scientist, yet nearly every author who cited the report uncritically accepted the findings. Davies stated, "if anyone doubts the university contribution to medical knowledge, they should read the report of Comroe and Dripps, who conclude that two-thirds of the contributions responsible for the major advances in. . cardiopulmonary disease were from basic research."(9) Citing Comroe and Dripps, Friend has claimed, "Basic research, i.e., investigation not clinically oriented when initiated, pays off in terms of key discoveries that subsequently prove to be important in clinical advances almost twice as much as other types of research combined.(10) In Circulation, Ross has editorialized:
This list of examples could be extended indefinitely to emphasize the importance of basic research to clinical medicine, but what we need is a careful scientific examination of the process of discovery. Fortunately, such a study was conducted in 1976 by Julius Comroe and Robert Dripps. . .. Combat the anti-intellectualism of some physicians who look on research as something medical school faculty members do for entertainment. Emphasize the relationship between today's research and tomorrow's therapy.(11)
Why has Comroe and Dripps' study been accepted so readily? Many of the papers reviewed apparently cited Comroe and Dripps' data for political reasons: These papers discussed the need for generous public support of basic research. Basic scientists may feel particularly vulnerable to budgetary constraints because legislators and philanthropists may easily view their work as unlikely to improve public well-being.
To the lay public, including legislators, Comroe and Dripps' study seems to "prove" that basic research is vital to medical progress. First, the numerical conclusions give the appearance of objectivity, even though the numbers themselves were derived from subjective determinations, Second, the study involved years of work; its magnitude lends it credibility. Third, Comroe and Dnpps were respected physiologists who had made highly regarded contributions to cardiopulmonary science. Their standing in the scientific community probably helped legitimize their review of basic research.
Conclusions
In order to appropriately prioritize government research support, the value of different research methods must be accurately assessed. Due to a flawed methodology and the omission of vital data, the study by Comroe and Dripps provides no basis for judging basic science's actual worth or determining which types of basic research most deserve support. The matter still awaits thorough and objective investigation.
The widespread use of Comroe and Dripps' report by scientists, who should readily discern its flaws, indicates that many scientists may be sacrificing professional ethics to political expediency. As science becomes increasingly technical and inaccessible to the public, there is a growing reliance on "experts." Such experts' failure to acknowledge obvious scientific shortcomings, like those in Comroe and Dripps' study, signals an urgent need for greater public skepticism and scrutiny regarding the "scientific" enterprise.
Acknowledgements:
I would like to thank Constance Young, Betsy Todd, Joan Dunayer, and Brandon Reines for their helpful review of this manuscript.
References
1. Comroe JH, Dripps RD: Scientific basis for the support of biomedical science. Science, 1976;192:105-1 11.
2. Comroe JH, Dripps RD: The Top Ten Clinical Advances in Cardiovascular-Pulmonary Medicine and Surgery 1945-1975. Washington DC: US Department of Health, Education, and Welfare, 1977.
3. Comroe JH, Dripps RD: Ben Franklin and open heart surgery. Circ Res 1974;35: 661-669.
4. Comroe JH: Answers to a congressman's questions. Circ Res 1969;25:501-503.
5. Comroe JH, Dripps RD: Artificial respiration, JAMA, 1946;130:381-383.
6. Comroe JH, Fowler WS: Lung function studies. VI. Detection of uneven alveolar ventilation during a single breath of oxygen: a new test of pulmonary disease. Am J Med 1951;10:408-413.
7. Black D: Looking at research. Lancet 1976;2:780-784.
8. Smith R: Comroe and Dripps revisited. Br J Med 1987;295: 1404-1407.
9. Davies TF: The NHS is dead: Long live the NHS. Br Med J 1976;2:1376-1378.
10. Friend C: The coming of age of tumor virology: Presidential address. Canc Res 1977;37:1255-1263.
11. Ross RS: The next 30 years -- will the progress continue? Circulation, 1980;62:1-7.