Skip to main content
Springer Nature Link
Log in

Defining the unscholarly publication: a bibliometric study of uncited and barely cited publications

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Bibliometric studies often exclude documents with little or no scientific content. Yet, identifying and classifying the unscholarly publication is a complex matter, and misclassifications often occur. Reference-based classifications are frequently proposed and implemented in bibliometric studies. Unfortunately, with little support for the actual classifications. In search of valid separation of scholarly/unscholarly publications, this study explores the correlation between number of references and citations received. Data was drawn from Scopus, and two separate analyses were conducted. The first analysis focus on journal articles published in the year 2000 from each of the four major subject areas in Scopus: Health sciences, life sciences, physical sciences and social sciences. The second analysis focus on all journal articles published in either 2000, 2005, 2010, 2015 or 2020. None of the analyses identified a natural cutoff point between scholarly and unscholarly publications. There is no significant point in the distribution that could more clearly distinguish scholarly from unscholarly publications than prior suggestions. In addition, the results of this study demonstrate how bibliometric analyses are impacted by defining scholarly publications on the number of references across different subject areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Ahlgren, P., Colliander, C., & Sjögårde, P. (2018). Exploring the relation between referencing practices and citation impact: A large-scale study based on Web of Science data. Journal of the Association for Information Science and Technology, 69(5), 728–743.

    Article  Google Scholar 

  • Al-Qallaf, C. (2003). Citation patterns in the Kuwaiti journal medical principles and practice: The first 12 years, 1989–2000. Scientometrics, 56(3), 369–382.

    Article  Google Scholar 

  • Ayer, A. J. (1936). Language, truth and logic. Gollancz.

    Google Scholar 

  • Barrios, M., Guilera, G., & Gómez-Benito, J. (2013). Impact and structural features of meta-analytical studies, standard articles and reviews in psychology: Similarities and differences. Journal of Informetrics, 7(2), 478–486.

    Article  Google Scholar 

  • Barrueco, J. M., & Inglada, V. J. (2002). Reference linking in economics: The Citec project. Advances in Knowledge Organization, 8, 251–257.

    Google Scholar 

  • Bordons, M., Aparicio, J., & Costas, R. (2013). Heterogeneity of collaboration and its relationship with research impact in a biomedical field. Scientometrics, 96, 443–466.

    Article  Google Scholar 

  • Brendel, J., & Schweitzer, S. (2019). The burden of knowledge in mathematics. Open Economics, 2(1), 139–149.

    Article  Google Scholar 

  • Chen, C. (2012). Predictive effects of structural variation on citation counts. Journal of the American Society for Information Science and Technology, 63(3), 431–449. https://doi.org/10.1002/asi.21694

    Article  Google Scholar 

  • Congleton, R. D., Marsella, A., & Cardazzi, A. J. (2022). Readership and citations as alternative measures of impact. Constitutional Political Economy, 33(1), 100–114.

    Article  Google Scholar 

  • Diodato, V. (1984). Impact and scholarliness in arts and humanities book reviews: A citation analysis. Proceedings of the 47th ASIS Annual Meeting: (pp. 217-220).

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178, 471–479.

    Article  Google Scholar 

  • Harzing, A. W. (2013). Document categories in the ISI web of knowledge: Misunderstanding the social sciences? Scientometrics, 94(1), 23–34.

    Article  Google Scholar 

  • Haslam, N., Ban, L., Kaufmann, L., Loughnan, S., Peters, K., Whelan, J., & Wilson, S. (2008). What makes an article influential? Predicting impact in social and personality psychology. Scientometrics, 76(1), 169–185.

    Article  Google Scholar 

  • Hu, Z., & Wu, Y. (2014). Regularity in the time-dependent distribution of the percentage of never-cited papers: An empirical pilot study based on the six journals. Journal of Informetrics, 8(1), 136–146.

    Article  Google Scholar 

  • Hu, Z., Wu, Y., & Sun, J. (2018). A quantitative analysis of determinants of non-citation using a panel data model. Scientometrics, 116(2), 843–861.

    Article  Google Scholar 

  • Iqbal, W., Qadir, J., Hassan, S. U., Javed, R. T., Mian, A. N., Crowcroft, J., & Tyson, G. (2019). Five decades of the ACM special interest group on data communications (SIGCOMM) a bibliometric perspective. ACM SIGCOMM Computer Communication Review, 49(5), 29–37.

    Article  Google Scholar 

  • Kalyane, V., & Sen, B. (1995). Bibliometric study of the Journal of Oilseeds Research. Annals of Library Science and Documentation, 42(4), 121–141.

    Google Scholar 

  • Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.

    Google Scholar 

  • Lakatos, I. (1978). The methodology of scientific research programmes. Cambridge University Press.

    Book  MATH  Google Scholar 

  • Lipetz, B. A. (1999). Aspects of JASIS authorship through five decades. Journal of the American Society for Information Science, 50(11), 994–1003.

    Article  Google Scholar 

  • Liu, J., Chen, H., Liu, Z., Bu, Y., & Gu, W. (2022). Non-linearity between referencing behavior and citation impact: A large-scale, discipline-level analysis. Journal of Informetrics, 16(3), 101318.

    Article  Google Scholar 

  • Lockett, M. W., & Khawam, Y. J. (1990). Referencing patterns in C&RL and JAL, 1984–1986: A bibliometric analysis. Library and Information Science Research, 12(3), 281–289.

    Google Scholar 

  • Metz, P. (1989). A statistical profile of College and Research Libraries. College and Research Libraries, 50, 42–47.

    Article  Google Scholar 

  • Mittermeyer, D., & Houser, L. J. (1979). The knowledge base for the administration of libraries. Library Research, 1, 255–276.

    Google Scholar 

  • Moed, H. F., Van Leeuwen, T. N., & Reedijk, J. A. (1996). A critical analysis of the journal impact factors of Angewandte Chemie and the Journal of the American Chemical Society: Inaccuracies in published impact factors based on overall citations only. Scientometrics, 37(5), 105–116.

    Article  Google Scholar 

  • Myerson, R. B. (1999). Nash equilibrium and the history of economic theory. Journal of Economic Literature, 37(3), 1067–1082.

    Article  Google Scholar 

  • Nash, J.F. (1950). Non-cooperative games. (Ph.D. Dissertation, Princeton University, 1950).

  • Nicolaisen, J. (2002). The scholarliness of published peer reviews: A bibliometric study of book reviews in selected social science fields. Research Evaluation, 11(3), 129–140.

    Article  Google Scholar 

  • Nicolaisen, J., & Frandsen, T. F. (2019). Zero impact: A large-scale study of uncitedness. Scientometrics, 119(2), 1227–1254. https://doi.org/10.1007/s11192-019-03064-5

    Article  Google Scholar 

  • Nicolaisen, J., & Frandsen, T. F. (2021). Number of references: A large-scale study of interval ratios. Scientometrics, 126(1), 259–285. https://doi.org/10.1007/s11192-020-03764-3

    Article  Google Scholar 

  • Patsopoulos, N. A., Analatos, A. A., & Ioannidis, J. P. (2005). Relative citation impact of various study designs in the health sciences. JAMA, 293(19), 2362–2366.

    Article  Google Scholar 

  • Peritz, B. C. (1981). Citation characteristics in library science: Some further results from a bibliometric study. Library Research, 3, 47–65.

    Google Scholar 

  • Popper, K. (1959). The logic of scientific discovery. Routledge.

    MATH  Google Scholar 

  • Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics. https://doi.org/10.1007/s11192-012-0779-5

    Article  Google Scholar 

  • Schrader, A. M. (1985). A bibliometric study of the JEL, 1960–1984. Journal of Education for Library and Information Science, 25, 279–300.

    Article  Google Scholar 

  • Schrader, A. M., & Beswick, L. (1989). The first five years of PLQ, 1979–1984: A bibliometric analysis. Public Library Quarterly, 9, 3–23.

    Article  Google Scholar 

  • Solla Price, D. J. (1965). Networks of scientific papers. Science, 149(3683), 510–515.

    Article  Google Scholar 

  • Solla Price, D. J. (1970). Citation measures of hard science, soft science, technology, and nonscience. In C. E. Nelson & D. K. Pollock (Eds.), Communication among scientsts and engineers (pp. 3–22). Heath.

    Google Scholar 

  • Stephenson, M. S. (1993). The Canadian Library Journal, 1981–91: An analysis. The Canadian Library Journal, 18(2), 1–18.

    Google Scholar 

  • Thagard, P. R. (1978). Why astrology is a pseudoscience. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association. (pp. 223–234).

  • Thanuskodi, S. (2010). Bibliometric analysis of the journal Library Philosophy and Practice from 2005-2009. Library Philosophy and Practice, 1.

  • Thelwall, M., & Sud, P. (2022). Scopus 1900–2020: Growth in articles, abstracts, countries, fields, and journals. Quantitative Science Studies. https://doi.org/10.1162/qss_a_00173

    Article  Google Scholar 

  • Thomson Reuters. (1994). The Thomson Reuters impact factor. Available at: https://clarivate.com/webofsciencegroup/essays/impact-factor/

  • van Leeuwen, T., Costas, R., Calero-Medina, C., & Visser, M. (2013). The role of editorial material in bibliometric research performance assessments. Scientometrics, 95(2), 817–828.

    Article  Google Scholar 

  • Verma, N., & Tamrakar, R. (2009). Analysis of contributions to defence science journal. DESIDOC Journal of Library & Information Technology, 29(6), 39.

    Article  Google Scholar 

  • Vieira, E. S., & Gomes, J. A. (2010). Citations to scientific articles: Its distribution and dependence on the article features. Journal of Informetrics, 4(1), 1–13.

    Article  Google Scholar 

  • Visser, M., van Eck, N. J., & Waltman, L. (2021). Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. Quantitative Science Studies, 2(1), 20–41.

    Article  Google Scholar 

  • Webster, G. D., Jonason, P. K., & Schember, T. O. (2009). Hot topics and popular papers in evolutionary psychology: Analyses of title words and citation counts in Evolution and Human Behavior, 1979–2008. Evolutionary Psychology, 7(3), 147470490900700300.

    Article  Google Scholar 

  • Windsor, D. A., & Windsor, D. M. (1973). Citation of the literature by information scientists in their own publications. Journal of the American Society for Information Science, 24, 377–381.

    Article  Google Scholar 

  • Worthern, D. B., & Shimko, A. H. (1974). Letter to the editor. Journal of the American Society for Information Science, 25(1), 72–73.

    Google Scholar 

  • Zeitlyn, D. (2020). Cite, plagiarise, pass-off: deixis, bibliographic imposture and photography. Philosophy of Photography, 11(1–2), 121–132.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Design and Communication, University of Southern Denmark, Universitetsparken 1, 6000, Kolding, Denmark

    Tove Faber Frandsen

  2. Department of Communication, University of Copenhagen, Njalsgade 76, 2300, Copenhagen, Denmark

    Jeppe Nicolaisen

Authors
  1. Tove Faber Frandsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeppe Nicolaisen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tove Faber Frandsen.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Appendix

Appendix

See Figs. 4 and 5

Fig. 4
figure 4

Number of references in journal articles published in 2000 in the four subject areas and percentage of uncited journal articles (0 citations) on a logarithmic scale. The number of references is shown on the x-axis of each graph. The percentage of uncited journal articles is shown on the y-axis

Full size image
Fig. 5
figure 5

Number of references in journal articles published in 2000 in the four subject areas and percentage of barely cited journal articles (0–4 citations) on a logarithmic scale. The number of references is shown on the x-axis of each graph. The percentage of uncited journal articles is shown on the y-axis

Full size image

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Frandsen, T.F., Nicolaisen, J. Defining the unscholarly publication: a bibliometric study of uncited and barely cited publications. Scientometrics 128, 1337–1350 (2023). https://doi.org/10.1007/s11192-022-04610-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-022-04610-4

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy