Skip to main content


Scientific impact of studies published in temporarily available radiation oncology journals: a citation analysis

Article metrics


The purpose of this study was to review all articles published in two temporarily available radiation oncology journals (Radiation Oncology Investigations, Journal of Radiosurgery) in order to evaluate their scientific impact. From several potential measures of impact and relevance of research, we selected article citation rate because landmark or practice-changing research is likely to be cited frequently. The citation database Scopus was used to analyse number of citations. During the time period 1996-1999 the journal Radiation Oncology Investigations published 205 articles, which achieved a median number of 6 citations (range 0-116). However, the most frequently cited article in the first 4 volumes achieved only 23 citations. The Journal of Radiosurgery published only 31 articles, all in the year 1999, which achieved a median number of 1 citation (range 0-11). No prospective randomized studies or phase I-II collaborative group trials were published in these journals. Apparently, the Journal of Radiosurgery acquired relatively few manuscripts that were interesting and important enough to impact clinical practice. Radiation Oncology Investigations’ citation pattern was better and closer related to that reported in several previous studies focusing on the field of radiation oncology. The vast majority of articles published in temporarily available radiation oncology journals had limited clinical impact and achieved few citations. Highly influential research was unlikely to be submitted during the initial phase of establishing new radiation oncology journals.


Scientifc publishing is an important task for radiation oncologists pursuing an academic career (Holliday et al. 2014). Many of these physicians or physician-scientists undergo yearly rating, and are more or less forced to produce a certain number of publications or surrogate achievements such as impact factor (Holliday et al. 2013). Collaborative groups competing for research funding also feel the pressure of succeeding with clinical trials and publishing their results in prestigious journals (Nieder 2012). Many well-established journals with high impact factor have low acceptance rates of submitted manuscripts. Continuous increases in number of submissions and competition for space result in a need for additional publication channels. Publishing companies launching new journals face the challenges of attracting a sufficient number of scientifically sound manuscripts, achieving high visibility in search engines, and achieving indexing in databases such as PubMed. A rapidly increasing impact factor is also helpful for the continued success of new journals. During the second half of the 1990s, when the field was dominated by the “International Journal of Radiation Oncology, Biology and Physics” and “Radiotherapy and Oncology”, two attempts were made to establish additional specialty journals, “Radiation Oncology Investigations” and “Journal of Radiosurgery”. However, these traditional print journals without page charges did not succeed and were taken off the market after few volumes. We hypothesized that failure was caused by insufficient number of publications that were interesting and important enough to attract readers and impact clinical practice, and that this lack of appeal would be reflected by low number of citations for most published studies. In order to evaluate this hypothesis, patterns of citation for these two journals were analysed.

Materials and methods

A systematic search of the citation database Scopus (Elsevier B.V., by use of the function ‘document search’ was performed on 3rd of July 2014. All articles published in “Radiation Oncology Investigations” and the “Journal of Radiosurgery” were selected irrespective of subject area, document and article type (review, clinical study, experimental study, case report etc.). Then, these articles were ranked by number of citations (field ‘times cited’ in the Scopus citation database). As requested during manuscript review, we also performed comparisons to the “International Journal of Radiation Oncology, Biology and Physics”. The respective Scopus search was performed on 6th of February 2015.


During the time period 1996-1999 “Radiation Oncology Investigations” published 205 articles, which achieved a median number of 6 citations (range 0-116). Figure 1 shows the distribution of citations. References (Seymour & Mothersill 1997; Sheridan et al. 1997; Desai et al. 1998; Schmidt-Ullrich et al. 1999; Smith & Haffty 1999; Johnson et al. 1998; Monga et al. 1999; Stickle et al. 1999; Merrick et al. 1998; Roach et al. 1997; Wazer et al. 1999; Joschko et al. 1997; Chidel et al. 1999; Durand & Olive 1997; Peschel et al. 1999; Norman et al. 1997; Kramer et al. 1998; Leborgne et al. 1997; Epperly et al. 1999; Chancy et al. 1998; Prete et al. 1998; Fernandez-Vicioso et al. 1997; Kang & Suh 1999; Banasiak et al. 1999; Nathu et al. 1998; Haffty et al. 1997; Gieger et al. 1997) and Table 1 include the 25 most frequently cited articles (all had at least 30 citations and were published in volumes 5-7, most emanated from the USA). The most frequently cited article in the first 4 volumes achieved 23 citations (Teicher et al. 1996). As also shown in Table 1, the median number of authors was 5 (range 2-10). Both clinical, dosimetric and radiobiological research as well as reviews achieved at least 30 citations. The most common topics were radiobiology (n = 8) and prostate cancer (n = 6). Topics covered all technological developments of that decade, e.g. brachytherapy, stereotactic radiotherapy and intensity-modulated radiotherapy. No final results of randomized studies or phase I-II collaborative group trials were published. The most common type of research were retrospective clinical studies (n = 10, Table 1). The median number of patients in these studies was limited (40.5, range 3-123).

Figure 1

Patterns of citation of all articles published in Radiation Oncology Investigations (ROI) and the Journal of Radiosurgery (JRS).

Table 1 The most frequently cited articles published in “Radiation Oncology Investigations” (in order of citation frequency, references 4-30)

The “Journal of Radiosurgery” published 31 articles, all in the year 1999, which achieved a median number of 1 citation (range 0-11, Figure 1). References (Garell et al. 1999; Maire et al. 1999; Gibon et al. 1999; Solberg et al. 1999; Sanghavi et al. 1999) include the 5 most frequently cited articles (all had at least 5 citations and were published in volume 2). Both clinical and physics research as well as reviews achieved any citations. No randomized studies or phase I-II collaborative group trials were published.

The “International Journal of Radiation Oncology, Biology and Physics” published almost 2200 articles during the time period 1996-1999. These achieved a median number of 22 citations (range 0-1135). Overall, 9% of articles achieved at least 100 citations (p = 0.0002 in Chi-Square test comparing the three journals).


The objective of this review was to identify pattern of scientific publication in two radiation oncology journals which were only temporarily available. While Radiation Oncology Investigations covered the whole field of radiation oncology, the Journal of Radiosurgery had a much narrower focus. After arbitrary decisions about which database to search (only those providing citation numbers could be considered for the purpose of this review), we performed a comprehensive evaluation of published research and number of citations accumulated during more than a decade after publication of the final journal volume. These citation numbers can be considered mature because previous analyses demonstrated that citation rate is gradually increasing for the first years after publication, followed by rapid decline (except for landmark randomized studies, which change clinical practice) (Stringer et al. 2010; Kondziolka 2011). Citation rate of published articles was chosen to define the most important contributions. Articles with high numbers of citations are likely those that impressed other clinicians/scientists and had impact on clinical practice or future developments in the field (Shao et al. 2013). It should be noticed that searches in different databases will result in more or less variable citation counts and that the present results therefore provide only a snapshot. This has been illustrated, e.g., in a cohort study of 328 articles published in JAMA, Lancet, or the New England Journal of Medicine between October 1, 1999, and March 31, 2000 (Kulkarni et al. 2009). Total citation counts for each article up to June 2008 were retrieved from Web of Science, Scopus, and Google Scholar. Google Scholar and Scopus retrieved more citations per article with a median of 160 and 149, respectively, than Web of Science (median 122, p < 0.001 for both comparisons). Importantly, Web of Science, Scopus and Google Scholar produced quantitatively and qualitatively different citation counts.

In a previous study of radiosurgery for various conditions, 1.5% of all articles (time period 1951-2010) achieved more than 100 citations (Kondziolka 2011). This figure corresponds well to that of articles in Radiation Oncology Investigations (3 out of 205, 1.5%), while the “International Journal of Radiation Oncology, Biology and Physics” achieved significantly higher proportions during the same time interval (1996-1999; 9%). The Top 100 radiation oncology articles from the time period 1999-2001 achieved a median of 208 citations (range 121-1149) (Nieder et al. 2013a), i.e. more than any of the publications reviewed here. When comparing articles published during different time periods, sources of bias must be acknowledged (variable temporal patterns of citation, increasing use of online databases in recent years, better access to published articles).

An analysis restricted to German radiation oncology publications revealed that most citations per year since publication were recorded for meta-analyses and randomized phase III trials (Nieder 2012). Lower figures were recorded for review articles, non-phase III prospective clinical trials, and retrospective clinical studies. Another analysis demonstrated that pattern of publication of the most influential radiation oncology studies was dominated by only two scientific journals: the Journal of Clinical Oncology and the International Journal of Radiation Oncology Biology and Physics (Nieder et al. 2013a). Several newly launched journals (first issue after 1999) managed to attract highly cited articles (Lancet Oncology, Nature Reviews Clinical Oncology or Nature Reviews Cancer, which had rapidly increasing impact factors). Apparently, despite controversy around impact factors and optimal evaluation of research productivity and quality (Kanaan et al. 2011; Durieux & Gevenois 2010), researchers find it attractive and desirable to publish their most important radiation oncology related work in the top journals of the field. The present findings are in line with these considerations.

Highly cited work is unlikely to be published in the first volumes of new journals. Prospective randomized trials and non-randomized trials performed by influential collaborative groups are prefentially submitted to well-established journals. It has also been shown that five newly established oncology journals, in this case open access publications without print issues, all published less than 50 articles in their first annual volume (Nieder et al. 2013b). First after 4-5 years number of articles per volume increased sharply. At that time, the two journals analysed here were already taken off the market. During the time period in question (1996-1999), no other new radiation oncology journals entered the market. However, afterwards two successful journals were established (in 2006 “Radiation Oncology” (open access, no print issues); in 2011 “Practical Radiation Oncology” (print issues, sister journal to the “International Journal of Radiation Oncology, Biology and Physics”)). Regarding “Radiation Oncology”, the annual volumes 2006-2008 all consisted of less than 50 articles. In 2009, contents expanded to 71 articles, while the most recent volume (2014) featured approximately 300. On the one hand, one could hypothesize that open access fees might prevent authors from submitting to journals like “Radiation Oncology”. On the other hand, advantages such as rapid publication and unlimited distribution and access appear to outweigh financial considerations. Institutional membership or publication funds and waiver of fees for authors from countries with limited resources might also encourage researchers to publish in open access journals.


Highly cited research typically appears in relatively few well-established journals. The vast majority of articles published in temporarily available radiation oncology journals had limited clinical impact and achieved few citations, especially those accepted in the initial phase of marketing.


  1. Banasiak D, Barnetson AR, Odell RA, Mameghan H, Russell PJ (1999) Comparison between the clonogenic, MTT, and SRB assays for determining radiosensitivity in a panel of human bladder cancer cell lines and a ureteral cell line. Radiat Oncol Invest 7:77–85, Cited 30 times

  2. Chancy AW, Pollack A, McNeese MD, Zagars GK (1998) Adjuvant radiotherapy for phyllodes tumor of breast. Radiat Oncol Invest 6:264–267, Cited 34 times

  3. Chidel MA, Suh JH, Greskovich JF, Kupelian PA, Barnett GH (1999) Treatment outcome for patients with primary nonsmall-cell lung cancer and synchronous brain metastasis. Radiat Oncol Invest 7:313–319, Cited 41 times

  4. Desai J, Stock RG, Stone NN, Iannuzzi C, DeWyngaert JK (1998) Acute urinary morbidity following I-125 interstitial implantation of the prostate gland. Radiat Oncol Invest 6:135–141, Cited 103 times

  5. Durand RE, Olive PL (1997) Physiologic and cytotoxic effects of tirapazamine in tumor-bearing mice. Radiat Oncol Invest 5:213–219, Cited 40 times

  6. Durieux V, Gevenois PA (2010) Bibliometric indicators: quality measurements of scientific publication. Radiology 255:342–351

  7. Epperly MW, Bray JA, Esocobar P, Bigbee WL, Watkins S, Greenberger JS (1999) Overexpression of the human manganese superoxide dismutase (MnSOD) transgene in subclones of murine hematopoietic progenitor cell line 32D cl 3 decreases irradiation-induced apoptosis but does not alter G2/M or G1/S phase cell cycle arrest. Radiat Oncol Invest 7:331–342, Cited 34 times

  8. Fernandez-Vicioso E, Suh JH, Kupelian PA, Sohn JW, Barnett GH (1997) Analysis of prognostic factors for patients with single brain metastasis treated with stereotactic radiosurgery. Radiat Oncol Invest 5:31–37, Cited 34 times

  9. Garell PC, Hitchon PW, Wen BC, Mellenberg DE, Torner J (1999) Stereotactic radiosurgery versus microsurgical resection for the initial treatment of metastatic cancer to the brain. J Radiosurg 2:1–5, Cited 11 times

  10. Gibon D, Coste E, Vial S, Vasseur C, Rousseau J (1999) Stereotactic localization in medical imaging: A technical and methodological review. J Radiosurg 2:167–180, Cited 7 times

  11. Gieger M, Wu JK, Ling MN, Wazer D, Tsai JS, Engler MJ (1997) Response of intracranial melanoma metastases to stereotactic radiosurgery. Radiat Oncol Invest 5:72–80, Cited 30 times

  12. Haffty BG, Son YH, Wilson LD et al (1997) Bioreductive alkylating agent porfiromycin in combination with radiation therapy for the management of squamous cell carcinoma of the head and neck. Radiat Oncol Invest 5:235–245, Cited 30 times

  13. Holliday E, Fuller CD, Wilson LD, Thomas CR Jr (2013) Success breeds success: authorship distribution in the Red Journal, 1975-2011. Int J Radiat Oncol Biol Phys 85:23–28

  14. Holliday EB, Jagsi R, Thomas CR Jr et al (2014) Standing on the shoulders of giants: results from the Radiation Oncology Academic Development and Mentorship Assessment Project (ROADMAP). Int J Radiat Oncol Biol Phys 88:18–24

  15. Johnson CR, Silverman LN, Clay LB, Schmidt-Ullrich R (1998) Radiotherapeutic management of bulky cervical lymphadenopathy in squamous cell carcinoma of the head and neck: Is postradiotherapy neck dissection necessary? Radiat Oncol Invest 6:52–57, Cited 59 times

  16. Joschko MA, Webster LK, Groves J et al (1997) Enhancement of radiation-induced regrowth delay by gemcitabine in a human tumor xenograft model. Radiat Oncol Invest 5:62–71, Cited 43 times

  17. Kanaan Z, Galandiuk S, Abby M et al (2011) The value of lesser-impact-factor surgical journals as a source of negative and inconclusive outcomes reporting. Ann Surg 253:619–623

  18. Kang S, Suh JH (1999) Radiation therapy for neurosarcoidosis: Report of three cases from a single institution. Radiat Oncol Invest Cited 30 times:309–312, Cited 30 times

  19. Kondziolka D (2011) Citation measures in stereotactic radiosurgery: publication across a discipline. Stereotact Funct Neurosurg 89:56–61

  20. Kramer BA, Wazer DE, Engler MJ, Tsai JS, Ling MN (1998) Dosimetric comparison of stereotactic radiosurgery to intensity modulated radiotherapy. Radiat Oncol Invest 6:18–25, Cited 36 times

  21. Kulkarni AV, Aziz B, Shams I, Busse JW (2009) Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA 302:1092–1096

  22. Leborgne F, Fowler JF, Leborgne JH, Zubizarreta E, Chappell R (1997) Biologically effective doses in medium dose rate brachytherapy of cancer of the cervix. Radiat Oncol Invest 5:289–299, Cited 36 times

  23. Maire JP, Darrouzet V, Trouette R et al (1999) Fractionated radiation therapy in the treatment of cerebello-pontine angle neurinomas: 12 Years of experience in 29 cases. J Radiosurg 2:7–11, Cited 7 times

  24. Merrick GS, Butler WM, Dorsey AT, Walbert HL (1998) Influence of timing on the dosimetric analysis of transperineal ultrasound-guided, prostatic conformal brachytherapy. Radiat Oncol Invest 6:182–190, Cited 55 times

  25. Monga U, Kerrigan AJ, Thornby J, Monga TN (1999) Prospective study of fatigue in localized prostate cancer patients undergoing radiotherapy. Radiat Oncol Invest 7:178–185, Cited 57 times

  26. Nathu RM, Mendenhall WM, Parsons JT (1998) Merkel cell carcinoma of the skin. Radiat Oncol Invest 6:233–239, Cited 30 times

  27. Nieder C (2012) Highly cited German research contributions to the fields of radiation oncology, biology, and physics: focus on collaboration and diversity. Strahlenther Onkol 188:865–872

  28. Nieder C, Geinitz H, Andratschke NH, Grosu AL (2013a) Landmark studies in radiation oncology: has the pattern of publication changed? J Cancer Sci Ther 5:115–118

  29. Nieder C, Dalhaug A, Aandahl G (2013b) Correlation between article download and citation figures for highly accessed articles from five open access oncology journals. Springerplus 2:261

  30. Norman A, Ingram M, Skillen RG, Freshwater DB, Iwamoto KS, Solberg T (1997) X-ray phototherapy for canine brain masses. Radiat Oncol Invest 5:8–14, Cited 37 times

  31. Peschel RE, Chen Z, Roberts K, Nath R (1999) Long-term complications with prostate implants: Iodine-125 vs. palladium-103. Radiat Oncol Invest 7:278–288, Cited 39 times

  32. Prete JJ, Prestidge BR, Bice WS, Dubois DF, Hotchkiss LA (1998) Comparison of MRI- and CT-based post-implant dosimetric analysis of transperineal interstitial permanent prostate brachytherapy. Radiat Oncol Invest 6:90–96, Cited 34 times

  33. Roach M III, Faillace-Akazawa P, Malfatti C (1997) Prostate volumes and organ movement defined by serial computerized tomographic scans during three-dimensional conformal radiotherapy. Radiat Oncol Invest 5:187–194, Cited 50 times

  34. Sanghavi S, Skrupky R, Badie B, Robins I, Tome W, Mehta MP (1999) Recurrent malignant gliomas treated with radiosurgery. J Radiosurg 2:119–125, Cited 5 times

  35. Schmidt-Ullrich RK, Contessa JN, Dent P et al (1999) Molecular mechanisms of radiation-induced accelerated repopulation. Radiat Oncol Invest 7:321–330, Cited 81 times

  36. Seymour CB, Mothersill C (1997) Delayed expression of lethal mutations and genomic instability in the progeny of human epithelial cells that survived in a bystander-killing environment. Radiat Oncol Invest 5:106–110, Cited 116 times

  37. Shao H, Yu Q, Bo X, Duan Z (2013) Analysis of oncology research from 2001 to 2010: a scientometric perspective. Oncol Rep 29:1441–1452

  38. Sheridan MT, O’Dwyer T, Seymour CB, Mothersill CE (1997) Potential indicators of radiosensitivity in squamous cell carcinoma of the head and neck. Radiat Oncol Invest 5:186, Cited 112 times

  39. Smith BD, Haffty BG (1999) Molecular markers as prognostic factors for local recurrence and radioresistance in head and neck squamous cell carcinoma. Radiat Oncol Invest 7:125–144, Cited 63 times

  40. Solberg TD, Ford JM, Medin PM et al (1999) Reproducibility of frame positioning for fractionated stereotactic radiosurgery. J Radiosurg 2:57–64, Cited 5 times

  41. Stickle RL, Epperly MW, Klein E, Bray JA, Greenberger JS (1999) Prevention of irradiation-induced esophagitis by plasmid/liposome delivery of the human manganese superoxide dismutase transgene. Radiat Oncol Invest 7:204–217, Cited 55 times

  42. Stringer MJ, Sales-Pardo M, Nunes Amaral LA (2010) Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. J Am Soc Inf Sci Technol 61:1377–1385

  43. Teicher BA, Bump EA, Palayoor S, Northey D, Coleman CN (1996) Signal transduction inhibitors as modifiers of radiation therapy in human prostate carcinoma xenografts. Radiat Oncol Invest 4:221–230, Cited 23 times

  44. Wazer DE, Jabro G, Ruthazer R, Schmid C, Safaii H, Schmidt-Ullrich RK (1999) Extent of margin positivity as a predictor for local recurrence after breast conserving irradiation. Radiat Oncol Invest 7:111–117, Cited 43 times

Download references

Author information

Correspondence to Carsten Nieder.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

CN participated in the design of the study and performed the statistical analysis. HG, NHA and ALG conceived of the study, and participated in its design and helped to draft the manuscript. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark


  • Radiation oncology
  • Radiotherapy
  • Research evaluation
  • Scientific publishing
  • Citation