Which of the following is a limitation of psycinfo compared to google scholar?

1Health Information Research Unit Department of Clinical Epidemiology and Biostatistics McMaster University Hamilton, Ontario, L8N 3Z5 Canada

Find articles by Angela May Eady

1Health Information Research Unit Department of Clinical Epidemiology and Biostatistics McMaster University Hamilton, Ontario, L8N 3Z5 Canada

Find articles by Nancy L. Wilczynski

1Health Information Research Unit Department of Clinical Epidemiology and Biostatistics McMaster University Hamilton, Ontario, L8N 3Z5 Canada

2Department of Medicine Faculty of Health Sciences McMaster University Hamilton, Ontario, L8N 3Z5 Canada

Find articles by R. Brian Haynes

Author information Copyright and License information Disclaimer

This study evaluated search strategies for finding high-quality studies on treatment and systematic reviews in PsycINFO.

64 journals were hand searched at McMaster University. Methodologic criteria were applied to clinically relevant articles to identify “pass” and “fail” articles. 4,985 candidate terms were compiled: 7,463 combinations for therapy articles and 5,246 combinations for reviews. Candidate search strategy results were compared with hand searches. The proposed strategies served as “diagnostic tests” for sound studies; the hand searches were the “gold standard.” Sensitivity, specificity, precision, and accuracy were calculated.

233 (32.5%) of 716 treatment articles met criteria for scientific merit, and 58 (11.5%) of 506 review articles met criteria for systematic reviews. For treatment studies, combined terms had a peak sensitivity of 97.9% (specificity 52.2%). Maximum specificity was 97.7% (sensitivity 51.5%). Sensitivity and specificity were each 79% when optimizing both while minimizing their difference. For review articles, combined terms had a peak sensitivity of 81.0% (specificity 54.4%). Maximum specificity was 98.1% (sensitivity 51.7%). Sensitivity and specificity were each 65% when optimizing both while minimizing their difference.

Empirically derived search strategies can achieve high sensitivity and specificity for retrieving sound treatment studies and review articles from PsycINFO.

Keywords: Databases, bibliographic, Psychological literature, Information retrieval

Clinicians increasingly use online access to evidence in the course of clinical care as well as for continuing education and research [1]. In addition, online searching is crucial for systematic reviewers who strive to find all available clinical studies that address their research question. However, information retrieval in electronic databases is often difficult and time-consuming. Relevant articles are scattered across a broad array of journals; the concentration of high-quality, relevant studies in a large database is dilute; the indexing in any large bibliographic database is limited; and databases contain many studies that are preliminary and not ready for clinical application. Added to these obstacles is the frequent lack of search skills [2].

Researchers have developed search strategies to assist clinicians and researchers with searching. Most of these strategies have been developed to find therapy and review articles in MEDLINE [3-9]. Some strategies for PsycINFO have also been developed and tested for specific interventions and study types (i.e., to find outcome studies of group psychotherapy [10] and randomized controlled trials of cognitive therapy for depression [11]), but these strategies are based on few journals and few target articles.

Learning how to use databases other than MEDLINE is important because unique studies can be found in specialized databases. The content area of PsycINFO pertains to health care: the behavioural causes and effects of physical disorders, health promotion and disease prevention, behavioural aspects of care, and behavioural treatments for physical illness [12]. Moreover, PsycINFO includes pharmacology articles that have a behavioural component (e.g., behavioural side effect), deal with compliance, or pertain to mental illness [12]. Although PsycINFO has some overlap with MEDLINE, it also provides access to unique articles. PsycINFO indexes a larger proportion (73%) of leading psychiatry journals than does MEDLINE (47%) [13]. Studies comparing database yields show that PsycINFO contributes studies on psychological and psychiatric topics that are not found in other databases [14] [15]. The validity of systematic reviews depends on the inclusion of all relevant studies, which requires searching more than one database [14]. The retrieval of relevant key studies is also important for competent clinical care. Thus, PsycINFO is an important source of studies for clinicians and researchers.

Strategies that test and evaluate strategies in PsycINFO are needed because this database uses a different thesaurus for indexing and MEDLINE terms cannot be used. Translating the MEDLINE index terms to PsycINFO vocabulary is most likely not effective because the database content is much broader (i.e., includes book chapters, books, and dissertations) and the database includes articles from fields outside of medicine (e.g., social work and psychology). Little information exists about how search terms perform; data showing how specific terms and their combinations perform in this specialized database are needed.

In the early 1990s, our group at McMaster University developed search filters for clinicians and researchers. These search filters were based on a small subset of 10 journals and covered 4 types of journal articles (therapy, diagnosis, prognosis and causation [etiology]) [16, 17]. This research was updated and expanded using data from 161 journals indexed in MEDLINE from the publishing year 2000 [18-21]. These search strategies have been adapted for use in the Clinical Queries interface of MEDLINE (//www.ncbi.nlm.nih.gov/entrez/query/static/clinical.html). Clinicians can easily access and use these search strategies by going to the Clinical Queries page in PubMed. We now report the extension of this research for PsycINFO, including the information retrieval properties of single terms and combinations of terms for maximizing the sensitivity and specificity of retrieving methodologically sound primary (original) studies on the treatment of health disorders, and those that are systematic reviews.

Although some searchers may wish to find both primary and secondary articles on treatment questions, we chose to present separate analyses for primary treatment studies and systematic reviews. The reasons for this decision are these: First, our review criteria apply to all reviews, not just those on treatment. Our systematic-review strategy also retrieves reviews on diagnosis, prognosis, etiology, clinical prediction, and economics questions. Second, a searcher may wish to retrieve only systematic reviews because they provide more useful information than single studies. Third, systematic reviewers may wish to conduct a review of reviews to determine how well a research question has already been studied. If a review exists, reviewers often need to search for studies published only after the existing review's search dates. Sensitive, primary study searches tend to be large and unwieldy, so a two-step approach saves time and resources. Our separate analyses will allow searchers the flexibility to search separately for studies and reviews.

We compared the retrieval performance of methodologic search terms and phrases in PsycINFO with a manual review of each article for each issue of 64 journal titles for the year 2000. Altogether, research staff hand searched 170 journal titles. These journals were chosen based on recommendations from clinicians and librarians, Science Citation Index Impact Factors from the Institute for Scientific Information, and the ongoing assessment of their yield of studies and reviews of scientific merit and clinical relevance for the disciplines of internal medicine, general medical practice, mental health, and general nursing practice (list of journals to be provided by the authors upon request). Of these 170 hand searched journals, 64 were indexed in PsycINFO; two of these 64 journals were not indexed in MEDLINE. Search strategies were developed using these 64 journals.

We compiled an initial list of search terms, including index terms and text words from clinical studies. Clinicians and librarians in the United States and Canada were consulted; we interviewed known searchers and made requests at meetings and conferences. A list was compiled of 4,985 unique search terms; of these, 2,583 returned results (the list of terms tested to be provided by the authors upon request). Search terms included text words, such as “random,” “blinded,” “controlled trial,” and “allocation concealment,” and index terms, such as “treatment effectiveness evaluation” and “treatment outcomes” (exploded—that is, including all of this term's narrower index headings that are grouped under the selected heading). For review articles, search terms included such text words as “effect size,” “literature search,” “search,” and “pooling,” and index terms, such as “literature review” and “automated information retrieval” (exploded).

As part of a larger study [22], research staff performance was rigorously calibrated before reviewing the journals, and inter-rater agreement for identifying the purpose of articles was 81% beyond chance (kappa statistic, 95% confidence interval [CI] 0.79 to 0.84). Inter-rater agreement for which articles met all methodologic criteria was 89% (CI 78% to 99%) beyond chance [22]. Six research assistants then hand searched all articles in each issue of the 64 journals and applied methodologic criteria to determine whether the article was methodologically sound for evaluation of a treatment or a review article. Thus, the treatment strategies were tested for their ability to retrieve articles about high-quality treatment studies from all other articles, including both low-quality treatment studies and all non-treatment studies. The methodologic criteria applied to studies of treatment were as follows: Random allocation of participants to comparison groups, at least 80% follow-up of those randomized, and analysis consistent with study design. The methodologic criteria applied to systematic reviews were as follows: Clearly stated topic of the review, explicit statement of the inclusion and exclusion criteria, description of the methods, and at least one of the primary studies included in the review was rigorous.

The proposed search strategies were treated as “diagnostic tests” for sound studies, and the manual review of the literature was treated as the “gold standard.” We determined the sensitivity, specificity, precision, and accuracy of each single term and combinations of terms in PsycINFO using an automated process. Borrowing from the concepts of diagnostic test evaluation and library science, sensitivity for a given topic is defined as the proportion of high-quality articles for that topic that are retrieved; specificity is the proportion of low-quality articles not retrieved; precision is the proportion of retrieved articles that are of high quality (and is equivalent to positive predictive value in diagnostic test terminology); and accuracy is the proportion of all articles that are correctly classified [23]. Confidence intervals (CIs) were calculated for estimates of sensitivity, specificity, precision, and accuracy using the Near exact (Miettinen) 95% CI method.

Individual search terms with sensitivity ≥ 10% and specificity ≥ 10% for a given purpose category were incorporated into the development of search strategies that included 2 or more terms. All combinations of terms used the Boolean OR—for example, “random.tw. OR blinded.tw.” The Boolean AND was not used because this strategy invariably compromised sensitivity, and our first goal was to ensure high retrieval of appropriate studies, with high specificity as a secondary goal. 7,463 search strategies were tested in the development of treatment search filters, and 5,246 search strategies were tested in the development of systematic review search strategies.

In addition to developing search strategies using the Boolean approach described above, we also evaluated the potential for improving performance using logistic regression. Two approaches were taken. First, we took the top performing Boolean search strategies and ORed additional terms to these base strategies using stepwise logistic regression. The level of significance for entering and removing search terms from the model was 0.05. Adding terms to the model stopped when the increase in the area under the ROC curve was < 1%. Second, we developed search strategies from scratch with stepwise logistic regression using these same cut off values. Both logistic regression approaches were compared with the Boolean approach to search strategy development when developing strategies for treatment articles and prognostic articles for MEDLINE. Treatment and prognosis were chosen because they represented the best and the worst cases for testing MEDLINE search strategy performance. For both purpose categories, the logistic regression approaches to developing search strategies did not improve performance compared with search strategies developed using the Boolean approach described above.[21] [24] Thus, for subsequent purpose categories, including systematic reviews and databases such as PsycINFO, the Boolean approach was used for search strategy development.

Indexing information was downloaded from PsycINFO for 6,301 articles from the 64 hand searched journals. Of these, 716 were classified as treatment, of which 233 (32.5%) were methodologically sound. For systematic reviews, 58 (11.5%) of the 506 articles classified as a review met the criteria. Search strategies were developed using all 6,301 articles. Additionally, strategies were tested for their ability to retrieve systematic reviews from all other articles.

Table 1 shows the best single term for high sensitivity (while keeping specificity at 50% or more) for finding treatment studies. The search term “random:” as a text word had 77% sensitivity and 94% specificity. (The colon is a symbol for truncation—all words that begin with “random” are included [e.g., randomly, randomized, and randomised]). The use of this term alone would miss a substantial number of relevant articles. Although the accuracy was high (93%), precision was modest (33%). Optimizing sensitivity and specificity (i.e., finding the best value for both while keeping the smallest possible difference between them) lowered the estimates further. Sensitivity reduced slightly to 75% and specificity was reduced to 68%; the precision was even lower at 8%, and accuracy was 68%.

Single Term with the Best Sensitivity (keeping Specificity ≥50%), Best Specificity (keeping Sensitivity ≥50%), and Best Optimization of Sensitivity and Specificity (based on the lowest possible absolute difference between sensitivity and specificity) for Detecting Studies of Treatment in PsycINFO in 2000. Values are percentages (95% confidence intervals).

OVID search*

(n = 233)

(n = 6068)

(n = 6301)

Table 2 shows the single-term strategies that had the best sensitivity, best specificity, and best optimization of sensitivity and specificity for finding systematic reviews. Exploding the index term “treatment” had sensitivity 55%, specificity 67%, precision 1.5%, and accuracy 67%.

Single Term with the Best Sensitivity (keeping Specificity ≥50%), Best Specificity (keeping Sensitivity ≥50%), and Best Optimization of Sensitivity and Specificity (based on the lowest possible absolute difference between sensitivity and specificity) for Detecting Systematic Reviews in PsycINFO in 2000. Values are percentages (95% confidence intervals).

OVID search*

(n = 58)

(n = 6243)

(n = 6301)

Combination of terms with the best results for sensitivity, specificity, and optimization of sensitivity and specificity are shown in Table 3 (treatment) and Table 4 (systematic reviews). For treatment, the use of multiple terms increased sensitivity and specificity, but the trade-off between the two was large. The most sensitive (98%) strategy (control:.tw. or random:.tw. or exp treatment) had a low specificity (52%), precision (7.3%), and accuracy (54%). An alternative strategy (placebo:.tw. or random:.tw. or exp treatment) showed that specificity can be increased (65%) through a small decrease in sensitivity (96%); the accuracy was 66% and precision was 9.6%. The most specific (97%) strategy (double-blind.tw. or random: assigned:.tw.) had a low sensitivity (52%) but high accuracy (96%); the precision was 47%. Again, an alternative strategy (double-blind.tw. or randomized.tw. or randomly assigned.tw.) increased sensitivity (77%), although specificity was slightly reduced (96%); the accuracy was still high (95%), and the precision was 41%. The strategy with the best optimization between sensitivity and specificity (double-blind.tw. or random: assigned.tw. or control:.tw.) had 79% sensitivity, 80% specificity, 13% precision, and 80% accuracy. Although the difference between sensitivity and specificity was reduced, a substantial number of studies was missed with a lower sensitivity.

Combination of Terms with the Best Sensitivity (keeping Specificity ≥50%), Best Specificity (keeping Sensitivity ≥50%), and Best Optimization of Sensitivity and Specificity (based on the lowest possible absolute difference between sensitivity and specificity) for Detecting Studies of Treatment in PsycINFO in 2000. Values are percentages (95% confidence intervals).

OVID search*

(n = 233)

(n = 6068)

(n = 6301)

Combination of Terms with the Best Sensitivity (keeping Specificity ≥50%), Best Specificity (keeping Sensitivity ≥50%), and Best Optimization of Sensitivity and Specificity (based on the lowest possible absolute difference between sensitivity and specificity) for Detecting Systematic Reviews in PsycINFO in 2000. Values are percentages (95% confidence intervals).

OVID search*

(n = 56)

(n = 6243)

(n = 6301)

Multiple-term strategies for systematic reviews showed a similar trade-off between sensitivity and specificity. The strategy (risk:.tw. or search:.tw. or exp treatment) with the best sensitivity (81%) had a specificity of 54% (precision 1.6% and accuracy 55%). When specificity was highest (98%), the strategy (meta-analysis.tw. or search:.tw.) had sensitivity of 51% (precision 20%; accuracy 98%). An alternate strategy (meta-analysis.tw. or effectiveness.tw. or search:.tw.) with a slight reduction in specificity (95%) increased sensitivity to 67% (precision 10%, accuracy 94%). Reducing the difference between sensitivity and specificity resulted in lower performance (sensitivity 66%, specificity 65%, precision 1.7%, and accuracy 65%).

Our study documents search strategies for use by clinicians and researchers that can help discriminate relevant, high-quality studies from lower quality studies of the treatment of health disorders and articles that are not about treatment. We also provide search strategies that help discriminate between high-quality systematic reviews and non-systematic reviews or systematic reviews that don't include at least one rigorous study. These methodological terms can be added to content terms to increase the likelihood of retrieving high-quality studies and reviews.

Watson and Richardson also developed treatment strategies for PsycINFO to find randomized controlled trials on cognitive therapy for depression [11] and outcomes studies on group psychotherapy [10]. Their strategies were based on hand searches of five to ten journals, incorporated particular content terms as well as methodologic terms, and did not report specificity estimates, so the results cannot be directly compared to those in our study. However, their results showed a lower sensitivity for PsycINFO than for MEDLINE. For example, their best-sensitivity strategy for finding randomized controlled trials of cognitive therapy for depression had a sensitivity of 97% for MEDLINE and 65% for PsycINFO [13].

Our study showed a similar difference, but only for the single-term strategy. The strategy with the best sensitivity (98% for therapy) used a combination of text word and index terms, and the sensitivity estimate for PsycINFO (98%) was much closer to the MEDLINE best-sensitivity estimate (99%) [24]. The question of whether this sensitivity is affected by the addition of content terms requires further study. Future research could also determine whether precision is improved by adding content terms.

Fewer articles indexed in PsycINFO than in MEDLINE passed the methods criteria (i.e., a smaller number of true positives). This smaller gold standard resulted in a wider CI for the best sensitivity estimate for the treatment strategy in PsycINFO (95% CI 96% to 99.7%) than for the treatment strategy in MEDLINE (95% CI 98.7% to 99.9%). However, the estimates of sensitivity from PsycINFO and MEDLINE were very similar, and the CIs substantially overlapped. Adding additional titles from the PsycINFO database would not have substantively altered this already favorable finding.

Our database for this study was not large enough to permit a “test-retest” approach. However, we have done this for treatment articles in the larger MEDLINE database and documented no important differences in performance (maximum difference 1.1% in 36 comparisons) [24].

The trade-off between sensitivity and specificity for treatment studies seems much larger for PsycINFO than for MEDLINE. For the best-specificity strategy, the sensitivity for PsycINFO was 52%, whereas for MEDLINE the sensitivity was 93% [24]. The National Library of Medicine has made a concerted effort to help searchers identify randomized controlled trials; it is probably no coincidence that the strategy with the best specificity for MEDLINE includes the publication type “randomized controlled trial” [24]; the use of this term boosts specificity while maintaining sensitivity and accuracy. It may be a useful publication type for PsycINFO also.

It is interesting that publication types do not feature in any of the PsycINFO strategies. Meta-analysis is a publication type in both MEDLINE and PsycINFO, and it featured in the high-sensitivity and top-precision strategies for MEDLINE [25] but not in PsycINFO. This verifies that the translation of terms from one database to another is not the best approach for devising search strategies, and it emphasizes the importance of these data showing the most effective terms for finding treatment studies or systematic reviews in PsycINFO.

The best-sensitivity estimate was substantially lower for systematic reviews (81%) than for treatment studies (98%). Better indexing could help improve the ability to find all relevant systematic reviews. For example, index terms or publication types that identify systematic reviews would be helpful. Although “meta-analysis” exists as an index term and publication type, the use of this term excludes any systematic review that does not include a meta-analysis, reducing the sensitivity of a search. However, our best-specificity strategy used meta-analysis as a text word, not as an index term or publication type (or all-field term), which raises questions about how consistently and accurately indexers apply “meta-analysis” to articles containing meta-analyses. Better performance for finding systematic reviews depends on accurate and consistent indexing as well as appropriate index terms.

PsycINFO differs from PubMed in that all the records are indexed when they enter the database. PubMed records are entered before indexing to provide earlier access, and can only be found by searching the titles and abstracts until indexing terms are added. In PsycINFO, all records are potentially retrievable by index terms and so improved indexing would yield a great benefit for searchers.

The strategies reported in this study include methodologic terms only. It would be useful to determine how they perform when added to content terms. This would also allow us to test the strategies of Watson and Richardson [10] [11] on a larger set of journals and target articles. We plan further studies to test the addition of content terms.

Selected combinations of indexing terms and text words can achieve high sensitivity or specificity in retrieving treatment studies and systematic reviews cited in PsycINFO. The reported search strategies will assist clinicians and researchers to retrieve relevant, high-quality articles. The strategies show a large trade-off between sensitivity and specificity when searching PsycINFO for treatment studies and systematic reviews. Consistent, accurate indexing using specific index terms for randomized controlled trials and systematic reviews could enhance performance (i.e., help maintain high sensitivity and high specificity).

This research was funded by the National Library of Medicine, USA. The Hedges Team includes Chris Cotoi, Aravin Duraikannan, Angela Eady, Brian Haynes, Susan Marks, Ann McKibbon, Doug Morgan, Cindy Walker-Dilks, Stephen Walter, Stephen Werre, Nancy Wilczynski, and Sharon Wong, all in the Department of Clinical Epidemiology and Biostatistics at McMaster University, Hamilton, Ontario, Canada.

Source of funding: National Library of Medicine, USA.

Conflict of interest statement

No conflicts of interest. All authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1. Westbrook JI, Gosling AS, Coiera E. Do clinicians use online evidence to support patient care? A study of 55,000 clinicians. J Am Med Inform Assoc. 2004;11:113–20. [PMC free article] [PubMed] [Google Scholar]

2. Ely JW, Osheroff JA, Ebell MH, Chambliss ML, Vinson DC, Stevermer JJ, et al. Obstacles to answering doctors' questions about patient care with evidence: qualitative study. BMJ. 2002;324:710. [PMC free article] [PubMed] [Google Scholar]

3. Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol. 2002;31:150–3. [PubMed] [Google Scholar]

4. Nwosu CR, Khan KS, Chien PF. A two-term MEDLINE search strategy for identifying randomized trials in obstetrics and gynecology. Obstet Gynecol. 1998;91:618–22. [PubMed] [Google Scholar]

5. Marson AG, Chadwick DW. How easy are randomized controlled trials in epilepsy to find on Medline? The sensitivity and precision of two Medline searches. Epilepsia. 1996;37:377–80. [PubMed] [Google Scholar]

6. Adams CE, Power A, Frederick K, LeFebvre C. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24:741–8. [PubMed] [Google Scholar]

7. Dumbrigue HB, Esquivel JF, Jones JS. Assessment of MEDLINE search strategies for randomized controlled trials in prosthodontics. J Prosthodont. 2000;9:8–13. [PubMed] [Google Scholar]

8. Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials. 1993 Doc No 33. [PubMed] [Google Scholar]

9. Shojania KG, Bero LA. Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. Eff Clin Pract. 2001;4:157–62. [PubMed] [Google Scholar]

10. Watson RJ, Richardson PH. Accessing the literature on outcome studies in group psychotherapy: the sensitivity and precision of Medline and PsycINFO bibliographic database searching. Br J Med Psychol. 1999;72:127–34. [PubMed] [Google Scholar]

11. Watson RJ, Richardson PH. Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of Embase, Medline and PsycINFO bibliographic databases. Br J Med Psychol. 1999;72:535–42. [PubMed] [Google Scholar]

12. American Psychological Association PsycINFO: Your source for Psychological Abstracts. Available from: //www.apa.org/psycinfo/

13. McDonald S, Taylor L, Adams C. Searching the right database. A comparison of four databases for psychiatry journals. Health Libr Rev. 1999;16:151–6. [PubMed] [Google Scholar]

14. Brettle AJ. Comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. Bull Med Libr Assoc. 2001;89:353–62. [PMC free article] [PubMed] [Google Scholar]

15. Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns. Complement Ther Med. 2004;12:228–32. [PubMed] [Google Scholar]

16. Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB. Assessment of methodologic search filters in MEDLINE; Proc Annu Symp Comput Appl Med Care; 1993. p. 5. [PMC free article] [PubMed] [Google Scholar]

17. Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994;1:447–58. [PMC free article] [PubMed] [Google Scholar]

18. Wilczynski NL, Haynes RB, for the Hedges Team Developing optimal search strategies for detecting clinically sound causation studies in MEDLINE; Proc AMIA Annu Symp; 2003. pp. 719–23. [PMC free article] [PubMed] [Google Scholar]

19. Wong SSL, Wilczynski NL, Haynes RB, Ramkissoonsingh R, for the Hedges Team Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE; Proc AMIA Annu Symp; 2003. pp. 728–32. [PMC free article] [PubMed] [Google Scholar]

20. Haynes RB, Wilczynski NL, for the Hedges Team Optimal search strategies for retrieving scientifically strong studies of diagnosis from MEDLINE. BMJ. 2004;1:328–1040. [PMC free article] [PubMed] [Google Scholar]

21. Wilczynski NL, Haynes RB. Developing Optimal Search Strategies for Detecting Clinically Sound Prognostic Studies in MEDLINE: An Analytic Survey. BMC Med. 2004;2:23. [PMC free article] [PubMed] [Google Scholar]

22. Wilczynski NL, McKibbon KA, Haynes RB. Enhancing retrieval of best evidence for health care from bibliographic databases: calibration of the hand search of the literature. Medinfo. 2001;10:390–393. [PubMed] [Google Scholar]

23. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. How to Practice and Teach EBM. 2nd edition Churchill Livingstone; Toronto: 2000. Evidence-Based Medicine; p. p73. [Google Scholar]

24. Haynes RB, McKibbon KA, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005;330:1179. [PMC free article] [PubMed] [Google Scholar]

25. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330:68. [PMC free article] [PubMed] [Google Scholar]