Journal of Criminal Justice 34 (2006) 147 – 152

What methods are most frequently used in research in criminology

and criminal justice?
Gary Kleck a,⁎, Jongyeon Tark b , Jon J. Bellows c
a
College of Criminology and Criminal Justice, Florida State University, 634 West Call Street, Tallahassee, FL 32306-1127, United States
b
Department of Criminal Justice, University of New Haven, 300 Boston Post Road, West Haven, CT 06516, United States
c
Juvenile Justice Educational Enhancement Program, Florida State University, 325 John Knox Road, Building L, Suite 102, Tallahassee,
FL 32303, United States

Abstract

Articles published in seven leading criminology and criminal justice journals were coded with regard to the research methods
used, focusing on the general research designs, data-gathering methods, and statistical analysis techniques employed. The results
indicated that survey research was by far the dominant mode of acquiring criminological information, that cross-sectional
nonexperimental designs still predominated, and that multivariate statistical methods were the norm. The findings could aid
criminology and criminal justice faculty in devising graduate methods curricula that reflected the state-of-the-art as currently
practiced by criminological researchers.
© 2006 Elsevier Ltd. All rights reserved.

Introduction commonly used by the scholars who publish in the

leading criminology and criminal justice journals.
What research methods are most commonly used by
scholars who study crime and society's reaction to Methods
crime? This question could be of general interest to
criminologists curious about the tools commonly used The focus was specifically on methods used by
by their colleagues. It is, however, also of special interest criminological researchers, so attention was confined to
to graduate faculty who must establish or periodically journals that were exclusively or primarily concerned
revise their curricula to keep their graduate training with criminological topics, rather than the wide array of
current. Some methodological skills may be of only very journals that occasionally publish such work but mostly
specialized interest to those focusing on narrow topics concern other subjects. Further, the focus was particu-
calling for rarely used techniques. Other techniques are larly on the methods used in the presumably better quality
so widely used as to be regarded as virtually required research published in the leading journals in the field.
topics in any graduate methods curriculum. The goal of There was some consensus, but no unanimity, about
this inquiry was to establish the research methods most which were the most important, prestigious or frequent-
ly cited journals. For example, Criminal Justice Review
appeared in the lists compiled by Sorenson, Patterson,
⁎ Corresponding author. Tel.: +1 850 894 1628; fax: +1 850 644 and Widmayer (1993), but not in rankings based on
9614. ratings of prestige in a survey conducted by Williams,
E-mail address: gkleck@mailer.fsu.edu (G. Kleck). McShane, and Wagoner (1992, summarized in Cohn,
0047-2352/$ - see front matter © 2006 Elsevier Ltd. All rights reserved.
doi:10.1016/j.jcrimjus.2006.01.007
148 G. Kleck et al. / Journal of Criminal Justice 34 (2006) 147–152

Farrington, and Wright (1998, p. 26)). Criminal Justice comments on articles previously published in the same
and Behavior was high in the ranks of journals assessed journals, and author replies to those comment articles
by Sorenson et al. (1993) and Williams et al. (1992), but were excluded. Each “case” in the analysis was an
not in the earlier assessments of Shichor, O'Brien, and original article.
Decker (1981). Likewise, Law and Society Review The senior author created a coding protocol (available
ranked high in lists generated by Regoli, Poole, and from that author) and determined the coding for any
Miracle (1982) but not in the citation-frequency ambiguous cases. The junior authors, graduate students in
rankings of Cohn and Farrington (1990). These criminology and criminal justice, coded the articles. Each
differences were probably partly due to actual changes article's journal, volume, issue number, and starting page
in importance over time and partly to different methods were recorded and each article was classified as to
of assessing importance. whether it was (a) nonempirical, which would include
Attention was concentrated on the “consensus” purely theoretical or conceptual papers, as well as
journals, those that had most consistently appeared in literature reviews with no original empirical evidence,
these lists. It was also important to cover any newer (b) empirical but exclusively descriptive, in the sense that
journals that could not be included in earlier lists there was no direct testing of causal hypotheses, or (c)
because the journals were created fairly recently. It empirical and analytical in the sense that it performed
would be undesirable for the selection of leading some tests of causal hypotheses. The analyses covered all
journals to be out of date, and thereby reflect prestige empirical articles appearing in these seven leading
confined to earlier eras. As it happens, one of the more journals in calendar years 2001 or 2002.
recent ranking efforts yielded a list of journals that met Empirical articles were coded as to their general
all these requirements. Williams and his colleagues research design. Did they use experimental methods
(1992) asked a sample of over 250 members of the (i.e., the work involved researcher manipulation of
Academy of Criminal Justice Sciences to rate, on a scale conditions to which subjects were exposed) or was
from 0 to 10, the prestige of the leading criminology and study purely nonexperimental? Did the research
criminal justice journals. The following journals were consider evidence concerning: (1) individual persons
found to be the seven most highly rated (ratings in only, (2) macro-level units (e.g., the populations of
parentheses): Criminology (CR, 8.7), Journal of Crim- cities or states), or (3) both individuals and macro-level
inal Law and Criminology (JCLC, 8.2), Justice units? Was the design (1) purely cross-sectional
Quarterly (JQ, 8.0), Journal of Research in Crime and (individuals or populations compared with each
Delinquency (JRCD, 7.9), Crime and Delinquency (CD, other), (2) purely longitudinal (a single population
7.6), Journal of Criminal Justice (JCJ, 7.3), and Journal studied at multiple points in time), or (3) both cross-
of Quantitative Criminology (JQC, 7.3). Taking into sectional and longitudinal (multiple populations studied
account the eras covered by various rating studies and at multiple points in time)?
the years when journals began to publish, all these Each empirical article was also coded for every data-
journals also consistently appeared in the top ranks of gathering technique that was originally used to generate
lists of leading journals in the field (Cohn & Farrington, the study's information. This would include methods
1990; Regoli et al., 1982; Shichor et al., 1981; Sorenson used by earlier researchers to gather information that
et al., 1993). Articles in these seven journals therefore was then subjected to secondary analysis by the authors
were covered. of the article being coded. Since articles often were
Readers might dispute the exact composition of based on more than one type of data, any one article
the list, and speculate about the resulting impact of could be coded for however many techniques happened
including or excluding particular journals that argu- to be used in the research. Survey-based articles could
ably should or should not have been included. also be coded affirmatively for more than one survey
Therefore, results will be presented for individual type. For example, the National Crime Victimization
journals, as well as for the entire set of seven taken Survey uses both face-to-face and telephone interview-
collectively, enabling readers to see for themselves ing, so articles using this source would be coded as
how methods vary across journals and how conclu- being based on both types of surveys.
sions might have differed had a given journal not Each article was coded for whether or not any of the
been included. following data-gathering techniques were used: (1)
The analysis encompassed all original articles and telephone interviews, (2) formal personal (face-to-face)
research notes in these seven journals. Editorial intro- interviews, (3) mail questionnaires, (4) other self-
ductions, speeches, letters, corrections, book reviews, administered (e.g., classroom) questionnaires, (5)
 G. Kleck et al. / Journal of Criminal Justice 34 (2006) 147–152 149

compilations of official statistics pertaining to macro- unusual, accounting for only 3 percent of empirical
level units (e.g., the Uniform Crime Reports), (6) articles. Further, most work done at either the individual
archival information drawn from existing records or macro-level of analysis was cross-sectional in nature:
concerning individual persons or events (e.g., sentencing 45 percent of articles were based on individual-level,
studies using court records, recidivism studies using purely cross-sectional analyses and another 15 percent
parole records, studies of violent crimes using police were based on macro-level purely cross-sectional
offense reports, etc.), (7) experimental observations, (8) analyses.
historical methods such as reading newspaper articles,
public speeches, diaries, memoirs, etc., (9) informal Data-gathering methods
interviews of the sort commonly conducted in field
research, (10) direct observation of behavior by The most striking finding concerns the data-gathering
researchers, and (11) miscellaneous other methods. methods used in this field. Survey research dominates
Articles were also coded as to whether the authors the field of criminology and criminal justice. No other
gathered their own data (primary data) or analyzed data method rivals survey methodology as a way of gathering
previously gathered by others (secondary data). This information on crime, criminals, and society's reaction
distinction is important because it bears on the im- to crime. Some kind of formal survey was used to
portance of learning how to locate and manage existing generate information used in 45 percent of all empirical
datasets. research articles published in the field's leading journals,
Finally, the empirical articles were coded for all compared to archival data, mostly drawn from criminal
statistical data analysis techniques employed. Once justice records, which was used in 32 percent of the
again, articles could be coded affirmatively for as articles, and official statistics concerning macro-level
many analytic techniques as were applied in the units, which were used in 26 percent of articles.
article. Articles were coded for (1) univariate descrip- Experimental research generated data for only 4 percent
tive statistics such as means, medians, and standard of the articles published in the seven journals. This
deviations, (2) bivariate statistics such as correlations would likely be different if journals oriented towards
and tabular measures of association, (3) ordinary least psychological research, such as Criminal Justice and
squares regression (OLS) and weighted least squares Behavior, had been included in the study. Nevertheless,
(WLS) regression, (4) logistic regression and other even taking this into account, it was unlikely that
techniques for analyzing limited dependent variables, experimental methods were used in more than a small
such as probit analysis, Tobit, etc., (5) structural fraction of crime studies.
equations modeling, including covariance modeling, Likewise, the principle tools used in qualitative field
analysis of latent variables, two-stage least-squares research, informal interviews and direct observation,
estimation, etc., (6) time series and interrupted time were used in only 12 percent of research. Even this
series methods such as ARIMA, Box Jenkins/Box- modest figure, however, overstated the prevalence of
Tiao methods, and variants thereof, (7) hierarchical research that was primarily qualitative in character,
linear modeling (HLM), (8) factor analysis, (9) event since many of the articles using these methods were
history and survival analysis, (10) Poisson and primarily quantitative articles that were supplemented
negative binomial models for analyzing count data, with a handful of informal interviews, such as statistical
and (11) miscellaneous other multivariate analysis analyses of sentencing patterns supplemented by a few
methods. interviews with judges. If a primarily qualitative article
was defined as one using informal interviewing or direct
Findings observation (including participant observation), but not
using multivariate statistical methods (i.e., articles using
Research designs multivariate methods were excluded), only 4.5 percent
of articles were qualitative. This prevalence did not vary
Table 1 displays the prevalence of various research substantially across journals.
designs and data-gathering methods used in criminol- Secondary analysis of existing datasets is central to
ogy and criminal justice research published in 2001– criminological research. Among empirical articles, no
02. Most (62 percent) work published in the leading less than 58 percent used secondary data. This might
journals was done at the individual level. Despite the reflect the difficulty and expense of original data
obvious advantages of situating individuals in their gathering. It also implies the importance of methodo-
macro-level contexts, multi-level research remained logical training in how to identify and obtain existing
150 G. Kleck et al. / Journal of Criminal Justice 34 (2006) 147–152

Table 1
General research methods used in criminological research
All seven journals CR JCLC JRCD CD JCJ JQC JQ
Total number of articles 375 63 43 33 55 93 30 58
Percent empirical 81.3 90.5 23.3 90.9 67.3 92.5 100 94.8

Research design (percent)a 305 57 10 30 37 86 30 55

Individual-level CX only 45.2 35.1 0.0 36.7 35.1 62.8 30.0 56.4
Individual-level CT only 13.8 5.3 20.0 23.3 10.8 12.8 30.0 10.9
Individual and macro-level 3.0 8.8 0.0 0.0 2.7 1.2 3.3 1.8
Macro-level, CX only 15.1 15.8 0.0 16.7 21.6 10.5 10.0 21.8
Macro-level, CT only 11.8 10.5 80.0 6.7 18.9 7.0 13.3 5.5
Macro-level, CX and CT 4.6 5.3 0.0 13.3 2.7 1.2 13.3 1.8

Data-gathering methods
Total number of empirical articles 305 57 10 30 37 86 30 55
Some kind of formal survey (percent)b 45.1 54.0 0.0 51.5 34.5 57.0 43.3 56.9
Telephone survey 9.0 12.5 0.0 7.1 5.6 9.3 13.3 7.3
Face-to-face survey 21.2 40.4 0.0 28.6 8.3 16.3 20.0 18.2
Mail survey 10.6 1.8 0.0 3.6 19.4 16.3 6.7 12.7
Non-mail self-administered survey 22.1 17.9 0.0 30.0 25.0 23.3 20.0 23.6
Official statistics 25.6 28.1 70.0 16.7 29.7 16.3 33.3 27.3
Archival data 31.8 28.1 70.0 20.0 48.6 26.7 23.3 36.4
Experimental data 4.3 1.8 0.0 3.3 5.4 5.8 0.0 7.3
Historical methods 3.0 1.8 10.0 0.0 2.7 3.5 0.0 5.5
Informal interviews 7.6 12.3 0.0 6.7 22.2 1.2 0.0 9.1
Direct observation 6.2 10.5 0.0 3.3 5.4 5.8 0.0 9.1
Qualitative researchb 4.5 6.3 0.0 6.1 5.5 3.2 0.0 8.6
CR = Criminology, JCLC = Journal of Criminal Law and Criminology, JRCD = Journal of Research in Crime and Delinquency, CD = Crime and
Delinquency, JCJ = Journal of Criminal Justice, JQC = Journal of Quantitative Criminology, JQ = Justice Quarterly, CX = cross-sectional, CT =
cross-temporal.
a
Of empirical articles, percent that used.
b
Informal interviewing or direct observation, but no statistics beyond univariate.

datasets relevant to one's research topic, as well as how fact the more advanced methods are often used to deal
to manage and analyze data gathered by others. with very specialized problems, such as two-way
causation, dependent variables with censored distribu-
Data analysis techniques tions, or nonindependent errors. The methods are
essential to dealing with the particular problems to
Table 2 displays the findings on the prevalence which they are applied, but each of the problems
of the various statistical techniques used to analyze arises relatively infrequently. Thus, while it is clear
criminological data. Nearly nine out of every ten that advanced statistical methods of some sort are
empirical articles used some kinds of univariate important tools in the field, it is not clear which
statistics, while 73 percent used some bivariate particular ones are the most important.
statistics, and nearly three-quarters used one type or Regarding differences across the seven journals,
another of multivariate statistics. The most common JCLC was the most deviant case. As a mixture of both
multivariate methods by far were OLS (or WLS) law review and criminology journal, it is perhaps not
regression and logistic regression or similar limited surprising that over three-quarters of its articles were
dependent variable methods. While the field is nonempirical, and its few (n = 10) empirical articles
undoubtedly moving towards more complex statistical were less likely to involve multivariate statistical
methods, each of the other multivariate methods was analysis. Conversely, articles in JQC showed the
used in less than 7 percent of the empirical articles. highest rate of use of advanced multivariate statistical
On the other hand, collectively speaking, one or more methods. These exceptions aside, the journals were
of the methods more advanced than OLS were used in more similar than different in the array of methods
50 percent of the articles. This is a reflection of the used in their empirical articles. In any case, since the
 G. Kleck et al. / Journal of Criminal Justice 34 (2006) 147–152 151

Table 2
Data analysis techniques used in empirical criminological research
All seven journals CR JCLC JRCD CD JCJ JQC JQ
Total number of empirical articles 305 57 10 30 37 86 30 55
Univariate statistics (percent)a 88.9 82.5 80.0 73.3 94.6 91.9 90.0 96.4
Bivariate statistics 73.4 86.0 30.0 86.7 73.0 66.3 56.7 81.8
Some kind of multivariate statistics 72.5 86.0 20.0 76.7 67.6 59.3 96.7 76.4
OLS/WLS 31.8 38.6 10.0 20.0 35.1 31.4 36.7 30.9
Limited dependent variable methods 28.5 36.8 10.0 30.0 35.1 16.3 30.0 36.4
Poisson/negative binominal 4.6 10.5 0.0 3.3 0.0 2.3 16.7 0.0
Structural equation modeling 6.6 10.5 0.0 23.3 0.0 2.3 3.3 7.3
Hierarchical linear modeling 5.6 14.0 0.0 6.7 2.7 1.2 6.7 5.5
ARIMA 2.3 3.5 0.0 3.3 5.4 0.0 3.3 1.8
Factor analysis 4.3 3.5 0.0 10.0 5.4 7.0 0.0 0.0
Event history/survival analysis 0.7 0.0 0.0 0.0 2.7 0.0 0.0 1.8
Multivariate beyond OLS/WLS 50.2 66.7 10.0 63.3 43.2 32.6 73.3 52.7
Multivariate beyond OLS/limited DVb 25.9 40.4 0.0 36.7 16.2 17.4 43.3 20.0
CR = Criminology, JCLC = Journal of Criminal Law and Criminology, JRCD = Journal of Research in Crime and Delinquency, CD = Crime and
Delinquency, JCJ = Journal of Criminal Justice, JQC = Journal of Quantitative Criminology, JQ = Justice Quarterly, OLS = Ordinary least squares,
WLS = weighted least squares, ARIMA = autoregressive integrated moving average.
a
Percent of empirical articles that used.
b
Multivariate statistics beyond OLS/WLS or limited dependent variable methods.

journals each averaged less than forty-four articles squares and logistic regression. These courses could
over the two-year period, any further conclusions then be followed up by more individualized elective
about detailed differences across journals are likely to coursework tailored to the particular needs of each
be unstable. student, focusing in depth on whichever advanced
statistical methods or specialized data-gathering strate-
Implications for graduate training in criminology gies were appropriate. Regarding statistical methods,
and criminal justice methods one option would be a course surveying three or four of
the relatively more frequently used advanced statistical
If doctoral students are to be properly trained to carry topics, such as structural equation modeling, hierarchi-
out research, or to be able to critically read and assess cal linear modeling, factor analysis, and perhaps Poisson
the research of others, doctoral methods curricula will regression. The topics covered might be rotated from
need to cover the research methods most commonly one year to the next, to keep pace with changes in the
used in their field. The choice of courses for doctoral field, and also to provide exposure to a larger number of
students is individualized to some degree, being advanced methods by allowing students to repeat the
composed of two components: (1) required or standard course when the topics changed. Other options would be
courses that all doctoral students take, because they are either separate courses on each topic, which would be
likely to be useful for most scholars in the discipline, more thorough but also more time-consuming, or
and (2) optional courses of more specialized value to directed independent study by the student, which can
research in the areas that a given individual student be hard on both students and faculty.
plans to explore, in either a dissertation or other work. A casual survey of the websites of the leading
The findings suggest some ways in which the required doctorate-granting programs in the field, however,
methods curriculum should go beyond the fairly indicates that it is unusual for graduate methods
standard “omnibus” methods course that covers all the curricula to include courses specifically devoted to
commonly used techniques, but fairly superficially. A either survey research or the problems of working
curriculum that accurately reflects the nature of the with criminal justice aggregate statistics and individ-
discipline would also give special emphasis to survey ual archival records. While these topics are common-
research methods, as well as the use and limitations of ly covered in an omnibus methods course, it is
official statistics and criminal justice archival records. It unlikely that such coverage can be as detailed and
would also provide required statistical course work that thorough as their prominence in the field would seem
covered univariate and bivariate statistics, ordinary least to justify.
152 G. Kleck et al. / Journal of Criminal Justice 34 (2006) 147–152

References Shichor, D., O'Brien, R., & Decker, D. L. (1981). Prestige of journals
in criminology and criminal justice. Criminology, 19, 461−469.
Cohn, E. G., & Farrington, D. P. (1990). Differences between British Sorenson, J. R., Patterson, A. L., & Widmayer, A. G. (1993).
and American criminology: An analysis of citations. British Measuring faculty productivity in a multidisciplinary field.
Journal of Criminal Justice Education, 4, 193−196.
Journal of Criminology, 30, 467−482.
Cohn, E. G., Farrington, D. P., & Wright, R. A. (1998). Evaluating Williams, F.P., III, McShane, M.D., & Wagoner, C.P. (1992). The
criminology and criminal justice. Westport, CT: Greenwood relative prestige of criminal justice and criminology journals.
Paper presented at the annual meetings of the American Society of
Press.
Regoli, R. M., Pole, E. D., & Miracle, A. W., Jr. (1982). Assessing the Criminology, New Orleans, LA.
prestige of journals in criminal justice: A research note. Journal of
Criminal Justice, 10, 57−67.