HOW TO INCREASE CITATIONS TO LEGAL
SCHOLARSHIP
ROB WILLEY
*
& MELANIE KNAPP
*
Abstract
Using nearly 250,000 law review articles published on HeinOnline over
a five-year period, the authors analyze citation patterns and their
relation to characteristics of the articles such as title length, number of
authors, article length, publication format, and more. The authors also
describe past citation studies and best practices in Search Engine
Optimization (SEO). The authors find that factors beyond article quality
likely impact scholarly citations. Drawing from the lessons in the
citation patterns, article characteristics, and SEO best practices, the
authors offer techniques to increase the article citation counts of
articles published in U.S. law journals. Using lessons from the SEO
world, the authors conclude with a detailed discussion of potential
problems with citation ranking schemes, such as citation cartels,
keyword stuffing, and perverse incentives for authors to avoid writing
on obscure but important legal topics.
*
[email protected], Faculty and Web Services Librarian, George Mason University Law
Library.
*
[email protected], Associate Director, George Mason University Law Library.
The Ohio State Technology Law Journal
158
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
CONTENTS
INTRODUCTION...................................................................... 160
OUR INTEREST IN CITATIONS ................................................... 161
LEVERAGING GOOGLE’S EXPERTISE FOR SCHOLARS .................. 162
“BLACK HAT” SEO & MANIPULATION OF U.S. NEWS’S SCHOLARLY
IMPACT ................................................................................... 164
TL;DR ........................................................................................ 166
METHODOLOGY ..................................................................... 168
CITATION BREAKDOWN ........................................................ 171
CHARACTERISTICS OF WELL-CITED ARTICLES ................ 173
CHARACTERISTIC ONE: LONG ARTICLES ................................... 174
SEARCH ENGINES AND SEO EXPERTS PREFER LONGER
CONTENT ............................................................................ 176
WHY LONGER CONTENT GETS MORE CITATIONS .................. 179
SUGGESTIONS TO LENGTHEN ARTICLES ............................... 184
EXAMPLE ARTICLES ............................................................ 187
ARE LONGER ARTICLES REALLY BETTER? ........................... 188
CHARACTERISTIC TWO: SHORT TITLES WITHOUT COLONS ........ 188
TITLE LENGTH ................................................................... 188
TITLE COLONS .................................................................... 189
ADDITIONAL ACADEMIC STUDIES .........................................190
SEARCH ENGINES AND SEO EXPERTS ALSO FAVOR SHORTER
TITLES ................................................................................ 193
NO SEO STUDIES ON USE OF COLONS .................................. 198
WHY ARE SHORTER TITLES WITHOUT COLONS PREFERRED?
.......................................................................................... 198
EXAMPLE TITLES ................................................................ 199
CHARACTERISTIC THREE: WRITE ON A POPULAR TOPIC ........... 200
SEO AND TOPIC SELECTION ............................................... 202
INCORPORATING POPULAR TOPICS ...................................... 203
FINDING POPULAR TOPICS ................................................. 204
COMPETITOR ANALYSIS ...................................................... 208
ADDITIONAL ITEMS TO CONSIDER.................................... 209
PUBLISH IN A WIDELY ACCESSIBLE JOURNAL .......................... 209
PUBLISH IN A TOP JOURNAL .................................................... 212
2021]
WILLEY & KNAPP
159
CITE YOURSELF? ..................................................................... 213
PUBLISH WITH A WELL-CITED AUTHOR ................................... 215
PUBLISH WITH A CO-AUTHOR? ................................................ 215
CHECK HEIN RECORDS AND ENSURE CREDIT FOR YOUR WORK
WITH ORCID ............................................................................ 216
MORE SEO PRACTICES TO CONSIDER....................................... 217
LESSONS FROM SEO FOR U.S. NEWS’S SCHOLARLY
IMPACT RANKING .................................................................. 221
FACTORS BEYOND QUALITY MAY IMPACT CITATIONS ............... 224
INCREASED PRESSURE LEADS TO STRONGER MOTIVES FOR ABUSE
...............................................................................................225
CITATION CARTELS ................................................................. 226
OPPORTUNITIES FOR TOP JOURNALS TO FAVOR THEIR SCHOLARS
.............................................................................................. 228
KEYWORD STUFFING .............................................................. 229
SEO EXPERTS & LAW SCHOOLS .............................................. 229
DECREASE IN IMPORTANT BUT LESS POPULAR TOPICS ............. 230
DO CITATIONS FOLLOW THE SCHOLAR? .................................. 232
PUBLISHING THE SAME CONTENT IN MULTIPLE PLACES .......... 232
LACK OF COVERAGE OF NON-LEGAL SOURCES ......................... 233
ISSUES WITH OCR TEXT RECOGNITION ................................... 234
“NO FOLLOW”: AVOID CITATIONS COUNTING WHEN YOU DO NOT
SUPPORT THE SCHOLARSHIP .................................................... 235
CONCLUSION ......................................................................... 236
160
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Introduction
Like many authors, we wrote this article because we want to share our
ideas, thoughts, and data. One way to verify that others are taking in
what we’ve shared is to monitor this article’s citation count. So, like
most scholars, we wrote this article to get cited.
In this article, we share characteristics that correlate to increased
citations in legal scholarship, with the goal of helping scholars get cited
more. We found these characteristics by looking at the differences
between articles that are well-cited and those that are not, searching for
characteristics that the data indicate will increase chances of citation.
Based on our findings, we created a set of guidelines to increase the
likelihood that an article will be cited. Our interest in this area grew out
of experience, working closely with faculty publications, and with
optimizing websites for Google’s search algorithm. That is why we use
SEO studies on Google’s algorithm as a source of comparison to many
of our findings.
When we began this study, we had theories on what would lead to
increased citation counts, primarily based on our personal preferences
and what we knew about Search Engine Optimization (SEO). In some
cases, the data aligned with our assumptions, but, in several instances,
the study’s results surprised us. They may surprise you too. Regardless,
we believe that if you are a scholar writing to get cited, you will benefit
from reading this article. At the very least, it will make you think
differently about how you write, structure, and publish your article. The
majority of the articles we reviewed have a legal focus, but many of the
findings align with similar studies in other fields.
1
1
See, e.g., Maarten van Wesel, Sally Wyatt & Jeroen ten Haaf, What a Difference a Colon
Makes: How Superficial Factors Influence Subsequent Citation, 98 SCIENTOMETRICS 1601
(2014) (finding, as we did, that longer articles tend to get more citations).
2021]
WILLEY & KNAPP
161
Over the course of gathering data, one thing that stood out is how many
articles have no citations. The total set we considered included 242,924
articles. From that set, 199,865 articles (or roughly 82 percent) had zero
citations. Don’t let that be you! What you have to say matters. Put in the
time, whether by using the suggestions in this article or ideas of your
own, to make sure people read and consider your work.
Perhaps the potential for increased emphasis on citations that U.S.
News’ proposed law school faculty scholarly impact ranking would add
also helped put the idea for this study on our minds. If done well, this
new ranking system has the potential to improve U.S. News’ influential
law school ranking. However, the proposal has a wide range of pitfalls.
Using the experience of other disciplines and the SEO world, we
conclude this article by highlighting some of these pitfalls. Our hope is
that drawing attention to these issues will allow the relevant parties to
learn from the mistakes others have already made.
Our Interest in Citations
Our library is responsible for uploading faculty papers to SSRN.
2
We
also help faculty with various research projects as they author articles.
Through our experience, we’ve seen how much download and citation
counts matter to faculty. This makes sense, as they are a factor in tenure
decisions and generally viewed as a measure of a scholar’s influence.
3
Given our closeness to the process, the importance of citations had been
2
SSRN FAQ Page, SSRN, https://www.ssrn.com/index.cfm/en/ssrn-faq/
[https://perma.cc/M9FE-ACTK] (“SSRN is a platform for the dissemination of early-stage
research.”).
3
Erin C. McKlernan et al., Use of the Journal Impact Factor in Academic Review, Promotion,
and Tenure Evaluations, ELIFE, Jul. 31, 2019, at 1; Dag W. Aksnes, Liv Langfeldt & Paul
Wouters, Citations, Citation Indicators, and Research Quality: An Overview of Basic
Concepts and Theories, SAGE OPEN, Feb. 7, 2019, at 1.
162
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
in the back of our minds for years. With U.S. News’ proposed ranking
system, the focus on citations will only increase.
4
Leveraging Google’s Expertise for Scholars
We started our study by thinking about why some articles gain
widespread popularity, while others do not. Experience with Search
Engine Optimization,
5
an industry that has grown out of website
owners’ desire to rank well in search results, led us to wonder whether
some of the principles SEO experts have identified would apply to
scholarly articles.
6
Ranking well on Google has similarities to being
well-cited. Most obviously, Google’s ranking algorithm considers the
websites that link to (i.e., cite) the webpages it ranks.
7
Additionally,
Google’s algorithm looks at factors that indicate whether the content it
4
Robert Morse, U.S. News Considers Evaluating Law School Scholarly Impact, U.S. NEWS
(Feb. 13, 2019, 1:00 PM), https://www.usnews.com/education/blogs/college-rankings-
blog/articles/2019-02-13/us-news-considers-evaluating-law-school-scholarly-impact
[https://web.archive.org/web/20200212212525/https://www.usnews.com/education/blogs/coll
ege-rankings-blog/articles/2019-02-13/us-news-considers-evaluating-law-school-scholarly-
impact].
5
For interested readers, here’s a little more on SEO. Google regularly crawls the internet for
content. It then creates an index of that content and runs it through an algorithm. How the
algorithm values or rates the indexed content determines where each webpage displays in
search results. The earlier a webpage appears in results, the more likely a user will access it.
Although underlying motivations may differ, the goal of most website owners is to increase
quality traffic to their websites, which means they want to be as close as possible to the top
position in search results. Google does not reveal how its algorithm works. Since there is no
authoritative source for what will lead to a high website ranking, people have speculated,
conducted studies, and done extensive research on what websites can do to increase their
chances of ranking well. Generally, these efforts to rank well in search results are referred to
as Search Engine Optimization or SEO.
6
Cf. Mary Whisner, My Year of Citation Studies, Parts 1-4, 110 L. LIBR. J. 167, 167-80, 283-
94, 419-28, 561-77 (2018) (Once we started research, we found that others had similar ideas.
For example, Mary Whisner investigated what leads an article to be well-cited). If you are
interested in another perspective on this topic, we recommend also taking a look at Whisner’s
work.
7
See What is SEO?, MOZ, https://moz.com/learn/seo/what-is-seo [https://perma.cc/VWD6-
VLJ4] (chart under How SEO works heading indicating that SEO professionals view link
related items as making up roughly 40 percent of Google’s ranking algorithm).
2021]
WILLEY & KNAPP
163
ranks provides the information people are seeking — similar to how a
citation indicates that the citing author found an article useful.
8
Here,
it’s important to remember that Google’s algorithm does not simply
look at the content itself — proper spelling, grammar, character count,
etcetera — Google also factors in indicators from actual users.
9
For
example, Google tracks how often users click an item when it appears
in their search results (the “click-through rate” in SEO jargon).
10
This
indicates how effective a page’s title and search snippet are. This data
signals people’s preference for one piece of content over another, and
Google uses this information to adjust its algorithm to better match its
users’ preferences. Scholarly articles also have titles, and the search
snippet has similarities to an article abstract. Given these and other
similarities, we thought there might be common ground between well-
cited articles and well-ranked websites.
Furthermore, there is a reason that Google’s search engine dominates
the search market.
11
Think about your own experience with Google
compared to a less popular search engine. In most instances, Google
provides superior results. Google gives users excellent results because
Google has spent countless dollars and hours honing and refining search
algorithms by analyzing the behavior of billions of users. Our study has
similar aims: we, like Google, want to find what types of content people
prefer and why. Of course, websites and scholarly articles are different,
and we consider those differences in our analysis.
8
See How Search Algorithms Work, GOOGLE,
https://www.google.com/search/howsearchworks/algorithms/ [https://perma.cc/HB2U-633K].
9
Id.
10
Brian Sutter, 7 User Engagement Metrics That Influence SEO, FORBES (Mar. 24, 2018, 5:32
PM), https://www.forbes.com/sites/briansutter/2018/03/24/7-user-engagement-metrics-that-
influence-seo/#6726bec6567b [https://perma.cc/2DT3-DVAX].
11
Estimates of Google’s share of the search engine market vary, but it’s clear that Google
dominates. See, e.g., Search Engine Market Share Worldwide Jan - Dec 2019, STATCOUNTER:
GLOBAL STATS, https://gs.statcounter.com/search-engine-market-share/all/worldwide/2019
[https://perma.cc/RCX2-N6WD] (putting Google’s market share at 91.89 percent in 2019).
164
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
As it turns out, most of the characteristics we identify as increasing the
chances of citation align with generally accepted SEO principles. This
led us to examine other SEO principles for items that we did not include
in our study but that may apply to legal scholars. Below we layout the
findings of our study and the corollary SEO principles, followed by
additional SEO principles that we did not study but may still have
relevance.
“Black Hat” SEO & Manipulation of U.S. News’ Scholarly Impact
Of course, applying SEO principles to legal scholarship has the
potential to impact articles in both positive and negative ways, as we
touch on in our concluding section. Today, most SEO experts include
helping their clients create the “compelling and useful content” that
Google says it wants as a key part of their services.
12
In the early days
of SEO, this wasn’t the case. Early iterations of Google’s algorithm
were much less advanced, and SEO experts quickly discovered they
could trick the algorithm to get their client’s sites to rank higher.
13
As
Google caught on to these tricks, it adjusted its algorithm and essentially
punished websites that used these tactics.
14
These adjustments have
made Google’s algorithm more difficult to deceive.
15
The increased
difficulty, combined with the risks of Google penalizing sites caught
attempting to manipulate its algorithm, has led most SEO experts to
12
Search Engine Optimization (SEO) Starter Guide, GOOGLE HELP: SEARCH CONSOLE HELP,
https://support.google.com/webmasters/answer/7451184?hl=en [https://perma.cc/8B3F-
83PQ].
13
Razvan Gavrilas, 44 Black Hat SEO Techniques that Will Tank Your Site, COGNITIVE SEO,
https://cognitiveseo.com/blog/12169/44-black-hat-seo-techniques/ [https://perma.cc/8BRV-
NP42] (illustrating that examples of this include: keyword stuffing, the practice of adding
keywords in a way that hurts the writing; paying for links to your site; and so on).
14
Id.
15
But see Roger Montti, SEO Contest Exposes Weakness in Google’s Algorithm, SEARCH
ENGINE J. (Nov. 13, 2018), https://www.searchenginejournal.com/google-algorithm-
loopholes/278093/
[https://web.archive.org/web/20200104220816/https://www.searchenginejournal.com/google-
algorithm-loopholes/278093/] (discussing how a website written in nonsensical strings of
Latin nearly won a ranking contest between SEO experts).
2021]
WILLEY & KNAPP
165
focus on making sites easy to use, easy for Google’s bots to access, and
filled with good content. As deceptive tactics fell out of favor, they
became referred to as “black hat” SEO.
16
Doing something with the intent of deception, even if the thing being
deceived is an algorithm, seems intrinsically wrong. Yet, attorneys can
likely understand the ethical dilemma that early SEO experts faced.
Like attorneys, SEO experts have a duty to advocate on behalf of their
clients. This includes the use of all legal methods. Most black hat SEO
tactics are not and have never been illegal, so many SEO experts viewed
using them as a duty to their clients. Further, black hat tactics became
so widespread that an expert who did not use these tactics risked letting
the sites they managed drop below inferior sites in the search result
rankings.
17
With the SEO background in mind, we wondered: will law schools
attempt to game and deceive the U.S. News and other ranking systems
the way early SEO experts gamed Google’s algorithm? They will have
similar motivations to those held by SEO experts, and the stakes for law
schools may even be higher. Plus, the early U.S. News system or other
scholarly impact ranking schemes may be particularly susceptible to
abuse, as Google’s algorithm was in its early days. Deans trying to
decide what to do may argue that not doing everything they can to
increase citation counts may lead to good schools dropping below peers
who choose to aggressively go after citations. Given these sorts of
arguments and the stakes involved, it seems likely that at least some
schools will try to manipulate the system. So, after our discussion on
ways to increase citations, we will briefly examine how schools and
scholars could manipulate the rankings.
16
What Is Black Hat SEO?, WORDSTREAM, https://www.wordstream.com/black-hat-seo
[https://perma.cc/EE83-YK75].
17
See A Brief History of SEO, HEROSMYTH (Apr. 20, 2020),
https://www.herosmyth.com/article/brief-history-seo [https://perma.cc/YYD9-WSMM].
166
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
To conclude the discussion of manipulative tactics, we want to point out
that the goal of this paper is not to create “black hat” tactics for legal
scholars. We do not recommend doing anything that will deceive
readers or algorithms. We recommend incorporating our suggestions
into your work only when you believe it can enhance the quality of your
work. There is a difference between purposefully manipulating a system
and applying lessons from the success of others to help your work have
a broader impact.
TL;DR
18
This article is on the longer side because the data indicates that longer
articles tend to be cited more often. However, we realize that many in
our target audience do not have time to read through and pull out all the
details. So, for those who just want a findings summary, we created a
simple chart.
19
18
TL:DR, MERRIAM WEBSTER, https://www.merriam-webster.com/dictionary/TL%3BDR
[https://perma.cc/YPF9-GRBD] (defining TL;DR as an informal abbreviation for “too long;
didn’t read.”).
19
We give full details on our dataset in the body of the paper, but a few notes may aid in
understanding the chart. Our study focused on three sets of articles from 2015-2019
(inclusive): one with the most-cited articles over that time period, a second with a subset of
articles with three citations, and a third where each article in the set has one citation and at
least one author with over 100 total citations to their name. We think the dataset with authors
that have over 100 total citations offers the best comparison to the top articles. Consequently,
the chart only includes the most-cited article data and the data from articles with one citation
written by at least one author with over 100 total citations.
2021]
WILLEY & KNAPP
167
Table 1: TL;DR Summary of Recommendations for Increasing
Citations
Recommendation Details More
Details
DO — Write long
articles
Top articles averaged 63 pages per
article.
The most frequently occurring page
lengths for top articles were 68 and 66
pages respectively.
79 percent of top articles were between
36-90 pages.
By comparison, less cited articles
averaged 27 pages, per article and 72
percent
ranged between 2
-
3
5
pages.
Page
174
DO — Keep titles
short
Top articles averaged 52 characters per
title.
The most frequently occurring title
lengths for top articles were 27 and 32
characters respectively.
Only 6.8 percent of top article titles had
over 100 characters.
By comparison, less cited articles
averaged 70 characters per title and 18
percent
had over 100 chara
cters per title.
Page
188
DON’T — use colons
in your title
Only 32 percent of top articles had a
colon in the title.
Comparatively, 55 percent of less cited
articles had a colon in the title.
Page
189
DO — Write on a
popular/timely topic
Articles on trending topics appear to garner
more citations per article than articles on other
topics.
Page
200
CONSIDER —
Publishing in widely
accessible journals
Limited data indicates that journals available
on Hein have more citations per article than
those with embargoes or not available.
Page
209
168
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
DO — Publish in a
top journal
37 percent of top articles were published
in one of 2018 W&L Law Journal
Rankings top-ten journals.
Only 4 percent of less cited articles were
published in one of 2018 W&L Law
Journal Rankings top
-
ten journals.
Page
212
CONSIDER —
Publishing with a co-
author
Our data only showed a slight difference in
number of authors per paper between different
segments, but other researchers have found
publishing with a co
-
author to be beneficial.
Page
215
DO
Read the rest of the article
.
Methodology
U.S. News’ proposal for its Scholarly Impact metric called for using
citations to articles available on HeinOnline and published in the
previous five years.
20
Since part of this paper looks at issues with U.S.
News’ proposal, and many law school deans and faculty are likely most
interested in any use of citations that may impact the U.S. News’
ranking, we used the five most recent years for our study. Further, the
databases, technologies, and expectations of authors change over time,
so what may have led to a high citation count fifteen or more years ago,
may not today.
21
We also chose to pull our data from HeinOnline (Hein)
20
Morse, supra note 4.
21
See Samuel D. Warren & Louis D. Brandeis, The Right to Privacy, 4 HARV. L. REV. 193
(1890) (the most cited article of all-time on HeinOnline, The Right to Privacy, does follow our
recommendation when it comes keeping the title short (just 20 characters), but it falls well
short of the ideal article length (28 pages, while the top articles in our study averaged 63
pages)). Also, below we discuss an older, similar study by Ian Ayres and Fredrick E. Vars. Ian
Ayres & Fredrick E. Vars, Determinants of Citations to Articles in Elite Law Reviews, 29 J.
LEGAL STUDS. 427, 440 (2000) (finding an optimal page length of 53 pages, less than our
study found optimal).
2021]
WILLEY & KNAPP
169
because U.S. News plans to use Hein’s data for its ranking
22
and because
Hein, arguably, has the best collection of legal journals of any
database.
23
Hein includes data for number of times accessed, articles
citations, and case citations. U.S. News plans to use “citations,
publications and other bibliometric measures”
24
for its ranking. Since
there are comparatively few case citations, we used only article citations
to rank the articles in our datasets.
When we pulled our data on the most-cited articles, Hein had 242,924
published articles from 2015-2019 (inclusive) in its Law Journal
Library.
25
Since Hein had more articles than we could reasonably
collect data on and analyze, we opted to focus on the top and bottom
ends of the entire set of data, hypothesizing that any trends would be
most pronounced at the extremes. Picking the top was easy: we simply
gathered data on the 500 most-cited articles between 2015-2019 (we’ll
refer to this sample as the “top-articles”). Choosing what to include in
the bottom proved harder. There were roughly 4000 articles with 3
22
Morse, supra note 4; see also Paul Caron, U.S. News to Publish Law Faculty Scholarly
Impact Ranking in 2021, TAXPROF BLOG (Nov. 9, 2020),
https://taxprof.typepad.com/taxprof_blog/2020/11/us-news-to-publish-law-faculty-scholarly-
impact-ranking-in-2021.html [https://perma.cc/GM96-X3AY] [hereinafter Caron, U.S. News
to Publish] (publishing an email stating that U.S. News plans to publish its first scholarly
ranking in 2021 and will use Hein for data).
23
See Bonnie J. Shucha, Representing Law Faculty Scholarly Impact: Strategies for
Improving Citation Metrics Accuracy and Promoting Scholarly Visibility, 40 LEGAL
REFERENCE S. Q. 81 (2021).
24
Morse, supra note 4.
25
Hein is constantly adding articles to its collection. On May 21, 2020, less than a month after
we completed our data gathering for this portion, the count had gone up to 245,973.
HEINONLINE (2020). Further, Hein notes that its library now includes material beyond
traditional law journals. HEINONLINE MARKETING DEPARTMENT, LAW JOURNAL LIBRARY,
https://heinonline.org/HeinDocs/LawJournalLibrary.pdf (“Though initially named the ‘Law’
Journal Library, this resource has grown from a small collection of law reviews to a
multidisciplinary journals database spanning more than 39 million pages. Its coverage is
comprehensive, beginning with the first issue ever published, and includes works from 60
different countries, as well as all 50 states and the District of Columbia. The nearly 3900
journals in the library span more than 1500 research subjects, including political science,
history, technology, religion, business, gender studies, psychology, and many more.”).
170
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
citations, by including every tenth article in this subset, we would have
around 400 articles, giving us a similar sample size to the top-articles
sample. We’ll refer to this sample as the “bottom-articles, 3 citations.”
While gathering the articles with 3 citations, we realized that a
significant number were written by authors with only 3 total citations,
likely indicating students who only authored a single note or
comment.
26
It didn’t seem fair to compare students writing a single
article as a journal requirement with faculty whose careers depend on
scholarship. And, although we didn’t analyze this, it also seems possible
there is a bias against citing student authored work. We also noticed that
on average articles in the bottom sample were slightly younger than
those at the top.
27
This makes sense for the simple fact that the older
articles had more opportunity to garner citations.
28
Since the top-end we
had chosen had two clear advantages over our first bottom sample, we
decided to do a second bottom sample. For the second version, we
looked at articles with one citation and only included articles from 2015
where at least one author had 100 or more total citations to their name
(we’ll refer to this sample as the “bottom-articles, author over 100
citations”). 2015 is the oldest year in our study, which meant that
articles collected under the new criteria had maximum time to get cited.
One hundred total citations may not seem like that many when Hein’s
most cited author has 28,279 total citations,
29
but these top authors are
the exception, in a class by themselves. To get a sense of this, of the
faculty Hein currently lists as being associated with Yale, 18 percent
26
Having so few citations indicates little incentive to gain them and, likely, very few
published works. Although other scenarios are possible, we think it most likely that these are
students who wrote a single note as part of a journal requirement.
27
Adding up all the years in each data set and calculating the average had the most-cited
articles coming in at 2015.73, while the set with three citations came in at 2016.29.
28
Ayres & Vars, supra note 21, at 430; see also Whisner, supra note 6, at 170 (noting that
four out of the five most cited works from the study came from the study’s earliest year).
29
Law Journals - Most-Cited: Authors, HEINONLINE, https://heinonline.org/ (last visited May
21, 2020) (as of May 21, 2020, Cass Sunstein (28,279 article citations) led Richard Posner
(24,155 article citations) by 4124 citations). These citations were compiled using Hein’s
authors profiles. From the databases landing page, navigate to “Law Journal Library”; choose
“LibGuide”; choose “Most-Cited”; scroll down and select “Authors.”
2021]
WILLEY & KNAPP
171
have under 100 total citations.
30
If 18 percent of the faculty at a top
school have under 100 citations, it becomes clearer that one hundred is
a significant number of citations, requiring a concerted effort by the
author. Adding this requirement eliminated most student authored
works. For these reasons, we believe our bottom-articles, author over
100 citations has better data. However, despite its shortcomings, we
decided to also include our findings on the original bottom-articles, 3
citations sample to provide an additional perspective.
For our top-articles sample, we wanted to capture a complete picture at
one moment in time. To do that, we gathered all our data for that subset
from April 9 – 15, 2020. While gathering the data, we monitored for
any changes to the ordering of the articles — none were observed. With
the bottom datasets, we were less concerned with capturing them at an
exact moment in time as both were only samples of a larger set and, as
long as they met the criteria we set, would suffice for our needs. We
compiled our bottom-articles, 3 citations sample between March 5 –
April 23, 2020, and our bottom-articles, author over 100 citations
sample between March 3 – April 8, 2020.
Our study focuses on correlation, not causation. With so many
variables, proving causation is very challenging. However, we believe
that correlation can give a strong indication of what’s working even
without proving causality.
Citation Breakdown
To gain context, the total population of law review articles from 2015-
2019 in Hein was 242,924 when we gathered our data. As detailed
above, we chose subsets of the top and bottom-most cited for our in-
30
We calculated these numbers with data from Hein’s author profiles. Author Profiles by
Institution: Yale Law School, HEINONLINE, https://heinonline.org/ (from the databases landing
page, navigate to “Law Journal Library”; choose “Author Profiles”; scroll down to access data
on Harvard and Yale).
172
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
depth analysis. We also grouped the entire set of 242,924 by number of
citations (i.e., how many articles had 10 citations, 11 citations, and so
on). The results surprised us. First, 199,865 (82 percent) of the articles
had zero citations.
31
Knowing how much time most authors invest in
writing these articles, we found this troubling, but also realized it
signaled a need for our article. Second, the number of citations needed
for an article to be considered a “well-cited” article within the past five
years, turned out to be lower than we had anticipated. Only the top 1.1
percent of all articles were cited ten or more times. Given this, it seems
reasonable to call any article with ten or more citations “well-cited.”
Further, it would certainly be fair to call an article with 24 citations (the
lowest number of citations any article in our top-articles sample had)
well-cited, especially since articles with 24 or more citations made up
only 0.2 percent of the total population.
31
Our percentages here vary a bit from an older study, where Thomas Smith found that 43
percent of articles are not cited at all, and about 79 percent get ten or fewer citations. Smith’s
study included articles much older than our five-year cap. So, this might indicate that there is
hope of citation for the articles not yet cited in our study—maybe citations will eventually
come their way. Or perhaps the citation network identified by Smith has grown even more
skewed in the time between his study and ours. Smith also included citations by cases, while
we only look at citations between articles. One final possibility is that the variation is due to
differences in coverage between LexisNexis, which Smith used, and HeinOnline. Thomas A.
Smith, The Web of Law, 44 SAN DIEGO L. REV. 309, 335-36 (2007).
2021]
WILLEY & KNAPP
173
Figure 1: Citation characteristics for the full population of 242,924
articles
Characteristics of Well-Cited Articles
With an idea of the bigger picture and some background on our
methods, we will now discuss our findings in detail with a focus on how
the information can help authors increase their citation counts and
whether our findings align with those in the SEO world. Specifically,
this section discusses the differences we identified between the articles
in the top-articles sample and the two bottom-article samples. All the
differences between the characteristics discussed in this section are
statistically significant based on t-tests run in Excel, unless otherwise
indicated.
43,059
17.7 %
1-23 times
199,865
82.3 %
0 times
529
0.2 %
"top
articles"
24 or more
times
2787
1.1 %
3316
1.4 %
"well
cited"
10 times
Citation Count Breakdown (number of times cited)
Total number of articles is 242,924 (100%)
174
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Others have studied correlations between easily measurable factors and
number of citations,
32
but, within the legal discipline there are
comparatively few citation studies.
33
Outside the legal field, there are
an abundance of studies looking at factors that influence citations.
34
Some of these studies considered data across multiple disciplines and
found that what can lead to citations in one discipline may not in
another.
35
For example, in some disciplines, longer titles do better,
while in others, shorter titles garner more citations (this is detailed
further below). The difference across disciplines means legal authors
should proceed with caution before relying on citation studies outside
the legal arena and keep the differences between disciplines in mind
when we discuss non-legal studies, including the SEO studies that we
focus on.
Characteristic One: Long Articles
Articles in the top-articles sample averaged 63 pages per article,
compared to 27 pages per article in the bottom-articles, author over
100 citations sample and 35 pages per article in the bottom-articles, 3
citations sample. Within the top-articles sample, the most frequently
occurring page lengths were 68 and 66 pages. As shown in Chart One:
Page Count Range and Frequency, the majority (57 percent) of
articles in the top-articles sample fell within 47-79 pages, with the vast
majority (79 percent) coming in between 36-90 pages. On the other side,
most (72 percent) of the articles in the bottom-articles, author over
32
E.g., Whisner, supra note 6, at 167.
33
Ayres & Vars, supra note 21. (Ian Ayres and Fredrick E. Vars produced the best study we
found, a well-done and thorough examination of factors that influenced citations of articles
published from 1980 to 1995 in the Harvard Law Review, Stanford Law Review, and The Yale
Law Journal).
34
See van Wesel et al., supra note 1, at 1602-04 (collecting studies on different factors that
increase citations across a wide range of disciplines and providing new research on various
factors that influence Sociology, General & Internal Medicine, and Applied Physics).
35
See, e.g., id. at 12.
2021]
WILLEY & KNAPP
175
100 citations were between 2-34 pages, with most (67 percent) of the
bottom-articles, 3 citations falling between 14-46 pages.
The longest article in the top-articles sample is 166 pages long and the
shortest comes in at 2 pages. Wide variations also appeared in both the
lower data sets: articles in the bottom-articles, author over 100
citations ranged from 1 to 94 pages and bottom-articles, 3 citations
had articles from 2 to 142 pages. Clearly, we have found trends, not
rules. The data establishes that you can write an article much shorter
than 63 pages and still get loads of citations. Likewise, you can write an
article over 60 pages and get few citations.
Figure 2: Pages per article (top articles only)
Our data shows that articles in our top-articles sample tended to be
significantly longer than those in our less-cited samples. Strengthening
this assertion, Ian Ayers and Fredrick Vars conducted a study much like
ours and found a similar correlation.
36
They looked at articles published
in Harvard Law Review, Stanford Law Review, and The Yale Law
36
Ayres & Vars., supra note 21, at 440.
176
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Journal between 1980 and 1995.
37
Like us, they found that longer
articles tended to receive more citations than shorter ones.
38
However,
their research identified 53 pages as the optimal length for citations, ten
pages less than the average of our top-articles sample.
39
We did not
seriously investigate why the optimal length Ayers and Vars identified
turned out shorter than what we found, but we hypothesize that the
optimal article length has increased over time. A quick review indicates
that we might be correct: the top-ten most-cited articles from 1900-1910
averaged 23 pages long, with no articles over 35 pages.
40
The difference
between our results and Ayers and Vars could also stem from differing
methods of data analysis. Regardless of the reason, both our study and
Ayers and Vars found that longer articles tend to be cited more than
shorter ones.
41
Therefore, we recommend that scholars strive to write
longer papers, at least 50 pages in length (close to what Ayers and Vars
found, within where most articles in our top-article sample fell, and
over the length of most papers in both our less-cited samples).
Search Engines and SEO Experts Prefer Longer Content
How do Google and SEO experts feel about article length? Before
answering, we need to make clear that what is considered a long blog
post or webpage differs significantly from what most people consider a
long law review article. In December 2013, 74 percent of Medium
articles were 825 words or less (or roughly one and a half pages single
37
Id. at 429.
38
Id. at 440.
39
Id.
40
Reasons for this (and verifying the hypothesis) are outside the scope of this piece but could
include: relatively higher printing costs in earlier eras, earlier articles discussing areas of the
law that had yet to develop all the complexities they later did, or a desire of newer authors to
write longer articles than their predecessors.
41
Studies from other fields have made similar findings. See, e.g., John Hudson, Be Known by
the Company You Keep: Citations—Quality or Chance?, 71 SCIENTOMETRICS 231, 234 (2007)
(finding that citations increase as page length does in a study of two economic journals).
2021]
WILLEY & KNAPP
177
spaced), with 94 percent under 1650 words (roughly 3 single spaced
pages).
42
With that context in mind, several studies indicate that Google and SEO
experts prefer longer content. First, Google appears to favor (i.e., rank
higher) longer content.
43
Second, a study of 912 million blog posts
found that longer content tends to get (a) more backlinks
44
and (b) more
social media shares.
45
Specifically, the study found that “long-form
42
Mike Sall, The Optimal Post Is 7 Minutes, MEDIUM: DATA LAB (Dec. 2, 2013),
https://medium.com/data-lab/the-optimal-post-is-7-minutes-74b9f41509b
[https://web.archive.org/web/20210217052851/https://medium.com/data-lab/the-optimal-post-
is-7-minutes-74b9f41509b] (“[o]verall, 74 percent of posts are under 3 minutes long and 94
percent are under 6 minutes long”). See also Read Time and You, MEDIUM: BLOG (June 3,
2014), https://blog.medium.com/read-time-and-you-bc2048ab620c
[https://web.archive.org/web/20210414144548/https://blog.medium.com/read-time-and-you-
bc2048ab620c] (“[r]ead time is based on the average reading speed of an adult (roughly 275
WPM). We take the total word count of a post and translate it into minutes. Then, we add 12
seconds for each inline image. Boom, read time.”).
43
See Matt Bentley, Data-Driven SEO Part 2: Does Long Content Really Rank Better?, CAN I
RANK, http://www.canirank.com/blog/does-long-content-rank-better/ [https://perma.cc/SRE2-
Y6KM] (discussing issues with other studies but still finding that longer content tends to rank
better than shorter content); see also Pandu Nayak, In-Depth Articles in Search Results,
WEBMASTER CENTRAL BLOG (Aug. 6, 2013),
https://developers.google.com/search/blog/2013/08/in-depth-articles-in-search-results
[https://perma.cc/U8Z3-82D6]; see also Dan Shewan, What Is Long-Form Content and Why
Does It Work?, WORDSTREAM: BLOG (Aug. 27, 2019),
https://webmasters.googleblog.com/2013/08/in-depth-articles-in-search-results.html
[https://perma.cc/A3VY-FBQJ] (stating that a Google search algorithm update was designed
to “help users find in-depth articles”).
44
Backlinks are links from another website to yours.
45
Brian Dean hypothesized that one reason longer content may rank better is because it has
more backlinks, one of the ranking signals used by Google. However, Bentley argues that long
content actually does not get more backlinks and long content ranks better simply because it
actually tends to be better. Bentley’s sample size is smaller than Dean’s. Regardless of who is
right on the backlinks point, both agree that longer content ranks better—only the reason is in
dispute. See Brian Dean, We Analyzed 912 Million Blog Posts: Here’s What We Learned
About Content Marketing, BACKLINKO (Feb. 19, 2019), https://backlinko.com/content-study
[https://perma.cc/PXY6-QC5F]; see also Shewan, supra note 43 (discussing WordStream’s
increased user engagement and traffic with the website’s switch to longer content and
178
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
content gets an average of 77.2 percent more links than short articles.”
46
It found that, while the preferred length is shorter for social shares than
backlinks, longer content still outperforms shorter content, with posts
of 1000 to 2000 words getting 56.1 percent more shares than ones with
less than 1000 words.
47
User preference for longer web content implies that most people using
Google
48
prefer longer content. We consider the increase in backlinks
and social shares especially relevant because backlinks (links from an
outside site to the studied site)
49
and social media shares are the web
equivalents of citations in academic journals, with backlinks being the
stronger corollary.
50
When an author cites another author, they usually
do so to support a claim they’ve made or, less often, to refute the other
author. Either way, the cited author has done enough to gain the
attention of their peers. Similarly, when a webpage links to another
additional studies finding benefits with longer content); see also Ideal SEO Content Length:
Flushing the Goldfish Cliché Down the Toilet, SWEOR (Mar. 5, 2020),
https://www.sweor.com/seocontentlength [https://perma.cc/8ZXZ-5HQR]. But see Bentley,
supra note 43 (finding, in a smaller study, that longer content does not bring more links).
46
Dean, supra note 45.
47
Id. But see Matthew Howells-Barby, The Anatomy of a Shareable, Linkable & Popular
Post: A Study of Our Marketing Blog, HUBSPOT (Sept. 16, 2015),
https://blog.hubspot.com/marketing/seo-social-media-study [https://perma.cc/D9RR-QGCL]
(finding that the longest articles got the most social shares in a study of 6192 blog posts in late
2015).
48
This article only looks at articles written in English and, consequently, also only considers
English SEO findings and recommendations, with a focus on the U.S.
49
Backlinks are also referred to as “inbound links” or “incoming links.” If a New York Times
article linked to a Wall Street Journal article, the Wall Street Journal article would have a
backlink from the New York Times article.
50
Interestingly, just as most of the articles in the period we looked at had zero citations, most
blog posts get zero backlinks or social shares. Dean, supra note 45 (“[t]he vast majority of
online content gets few social shares and backlinks. In fact, 94 percent of all blog posts have
zero external links.”).
2021]
WILLEY & KNAPP
179
page, it does so either to support or refute something said on the page.
51
Tweets and other social shares have similar purposes. The fact that both
long web pages and long legal articles get more attention than short ones
signals a preference in both areas for longer content. To sum up, the
SEO research aligns with what we found: longer content is generally
more likely to be cited. However, as we touched on above, our study,
the one by Ayres and Vars, and some SEO studies found the benefit of
length only extends so far.
52
Why Longer Content Gets More Citations
As to why longer content is more cited, we identified several possible
reasons. More content means an opportunity to cover more topics,
including tangential ones. If an article focused on antitrust has a well-
done section on law and economics, perhaps citations to the law and
economics portion push the article’s citation count past similar articles
that only discuss antitrust. Hein uses artificial intelligence to create a
list of topics for many articles in its Law Journal Library.
53
While
collecting data, we recorded and counted these topics. The top-articles
sample averaged 5.29 topics per article, while the bottom-articles,
author over 100 citations sample averaged 4.90 and the bottom-
articles, 3 citations 5.12.
54
The difference between the top-articles and
bottom-articles, author over 100 citations supports our inference that
51
Because links signal to Google support for the linked to website, savvy website owners will
tell the search engine when they do not actually support the content they are linking to. See
The Beginner’s Guide to SEO: Link Building & Establishing Authority, MOZ,
https://moz.com/beginners-guide-to-seo/growing-popularity-and-links
[https://perma.cc/R4U8-XFZS] (discussing follow vs. no-follow links and how they can be
used to avoid passing support to a linked to website).
52
See Dean, supra note 45 (finding that 1000-2000 words is the optimal length for social
media shares); see also, Sall, supra note 42 (finding 7 minutes, roughly 1600 words, to be the
optimal length for capturing the most reading time).
53
Lauren Mattiuzzo, Alexa: Research Artificial Intelligence in HeinOnline for Me,
HEINONLINE BLOG (Jan. 14, 2020), https://home.heinonline.org/blog/2020/01/alexa-research-
artificial-intelligence-in-heinonline-for-me/ [https://perma.cc/RS65-UABE].
54
Note that while the difference between 5.29 and 4.90 is statistically significant, the
difference between 5.29 and 5.12 is not.
180
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
covering more topics may increase an article’s citation count. Further,
another study of law review citations found that only two out of the 100
citations they reviewed cited the article to build on or respond to it.
55
The other 98 were either used to support a fact or opinion or for unclear
reasons.
56
If articles are mainly being cited for facts or opinions, an
article with factually accurate tangents has more chance of being cited
than one without.
Another possibility, offered by some SEO experts
57
and at least one law
professor,
58
is that readers equate length with authority or expertise.
This seems logical. In the legal arena, we tend to assume that a multi-
volume treatise is more authoritative than a single volume hornbook.
We also generally think books provide a greater level of detail than
articles, and so on. Since most authors want to support their assertions
with the most authoritative content available, it follows that they would
prefer longer articles to shorter ones when given the choice.
It may also be that readers consider longer articles easier to skim.
Personally, we find reading a dense, 100+ page article daunting and
often not worth the effort when we only need a small portion of it. Our
guess is that at least some other members of the legal community feel
similarly. For us, and like-minded scholars, scanning a long article to
quickly home in on what you need is often faster than reading an entire
short article. Further, perhaps authors of longer articles realize that
55
Jeffrey L. Harrison & Amy R. Mashburn, Citations, Justifications, and the Troubled State
of Legal Scholarship: An Empirical Study, 3 TEX. A&M L.R. 45, 74 (2015); see also James E.
Krier & Stewart J. Schwab, The Cathedral at Twenty-Five: Citations and Impressions, 106
YALE L.J. 2121, 2122 (1997) (stating that survey articles are cited frequently because they
conveniently convey facts).
56
Harrison & Mashburn, supra note 55.
57
Manick Bhan, Content Length and SEO: Does it Really Matter?, LINKGRAPH (Mar. 9,
2020), https://linkgraph.io/content-length-and-seo-does-it-really-matter/
[https://perma.cc/6HDE-87PE] (“[l]ong copy gives the impression of expertise, credibility,
and extensive knowledge on a topic.”).
58
See Scott Dodson, The Short Paper, 64 J. LEGAL EDUC. 667, 668 (2014) (arguing against
long papers but stating that legal scholars “use length as a proxy for the value of the work”).
2021]
WILLEY & KNAPP
181
increased length requires tighter organization and use additional reading
aids that make it easier to scan the content, like a table of contents and
plenty of headings. When it comes to shorter articles, some researchers
might feel guilty about not reading the entire thing and may fear missing
the part most important to them if they do not. When this is the case,
reading a “short” twenty-page article will usually take longer than
scanning a 100-page piece and only reading the five relevant pages.
Some SEO experts have drawn similar conclusions.
59
We did not gather
any data relevant to this hypothesis, but, as discussed further below,
authors of long articles should do their best to make skimming easy for
their readers.
Most SEO experts agree that the earlier a website appears in search
results, the more likely a searcher will view it.
60
It is very likely that the
top results in legal databases also get more traffic than the lower ones,
suggesting another possible reason longer articles are cited more often.
Depending on the database’s algorithm, longer articles may appear
higher in legal database search results, making them more likely to be
found and cited. Early versions of Google’s algorithm ranked content
higher partially based on the number of times a searched-for word
appeared on a website.
61
Google has since adjusted its algorithm, but
the legal databases most people use for law reviews have less
sophisticated search algorithms that may still use the total number of
times the searched for keyword appears as a factor in ranking results by
59
Neil Patel, How Long Should Your Blog Articles Be? (With Word Counts for Every
Industry), NEILPATEL: BLOG, https://neilpatel.com/blog/long-blog-articles/
[https://web.archive.org/web/20210421223524/https://neilpatel.com/blog/long-blog-articles/].
60
E.g., Kelly Shelton, The Value of Search Results Rankings, FORBES (Oct. 30, 2017, 8:00
AM), https://www.forbes.com/sites/forbesagencycouncil/2017/10/30/the-value-of-search-
results-rankings/#6bc667c144d3 [https://perma.cc/9B8U-BH3Z] (discussing studies that
found websites on the first page of Google’s search results get between 71 percent and 92
percent of clicks, with second page results getting only 6 percent).
61
See Megan Marrs, The Dangers of SEO Keyword Stuffing, WORDSTREAM: BLOG (Oct. 23,
2017), https://www.wordstream.com/blog/ws/2012/03/21/dangers-of-keyword-stuffing
[https://perma.cc/6P4C-JTLZ] (explaining that increasing the number of times a keyword
appeared on a website was once a way to rank better for that keyword).
182
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
relevance.
62
Additionally, the main legal databases (Westlaw and Lexis)
allow users to run searches that only display results where the entered
keyword occurs at least x number of times, favoring longer articles.
63
Longer articles also have more space and opportunity to work in related
keywords and synonyms that authors may search for, further increasing
their chances of appearing sooner in search results.
64
Again, our data
does not support or disprove this assertion, but it does seem possible
that part of the success longer articles enjoy stems from favorable
treatment by the legal database algorithms and search options.
While we suspect all legal database algorithms lag behind Google, some
do claim significant advances. For example, Westlaw has claimed to be
“the world’s most advanced legal search engine.”
65
For different
reasons, long articles could also excel within more advanced legal
search algorithms, as long webpages do in Google’s algorithm.
Google’s algorithm favors webpages that users spend more time on.
66
In the past, Westlaw has stated that its algorithm determines relevancy
62
Susan Nevelow Mart, The Algorithm as a Human Artifact: Implications for Legal
[Re]Search, 109 L. LIBR. J. 387 (2017).
63
Developing a Search with LexisNexis, LEXISNEXIS, https://www.lexisnexis.com/bis-user-
information/docs/developingasearch.pdf [https://perma.cc/G75W-VJYZ]; Westlaw Edge Tip:
What Is the “At Least” Function, and How Do I Use It?, THOMSON REUTERS,
https://legal.thomsonreuters.com/blog/westlaw-edge-tip-what-is-the-at-least-functional-and-
how-do-i-use-it/ [https://perma.cc/4XHH-87AU].
64
See How to Keep Keyword Density Natural and Avoid Keyword Stuffing, SEMRUSH: BLOG
(Apr. 15, 2013), https://www.semrush.com/blog/how-to-keep-keyword-density-natural-and-
avoid-keyword-stuffing/ [https://perma.cc/LLN3-5R43] (stating that the best way to
“naturally” avoid keyword stuffing is to increase the word count).
65
WestSearch: The World’s Most Advanced Legal Search Engine, WESTLAW,
http://info.legalsolutions.thomsonreuters.com/pdf/wln2/l-355700_v2.pdf
[https://perma.cc/U7NZ-25UR]; see also Mart, supra note 61, at 413-15 (finding that
Westlaw’s algorithm delivers the most relevant results among the legal databases studied).
66
Dean, supra note 45.
2021]
WILLEY & KNAPP
183
based on a variety of factors, including “usage by research
professionals.”
67
Ultimately, we cannot be sure how Westlaw’s proprietary algorithm
currently works, but it is possible that it has followed the industry
leader, Google, and factored in the amount of time users spend viewing
content. Long articles require more time to read, at least if you read all
the content (see above for an instance where that may not be the case).
Thus, a long article may fair better in an algorithm that factors in the
amount of time a user spends on a page. Additionally, Westlaw has
previously stated that it factors in “meaningful interactions” when
ranking and returning results.
68
The listed examples of “meaningful
interactions” Westlaw gives, include print, email, and downloads.
69
Pre-
COVID, research showed a preference for print when it comes to longer
sources.
70
Further, personal experience and logic dictate that users are
67
Compare WestlawNext Q&A Session: Refining Your Results, THOMPSON REUTERS: LEGAL
SOLUTIONS BLOG (June 18, 2010), https://blog.legalsolutions.thomsonreuters.com/legal-
research/westlawnext-qa-session-refining-your-results/
[https://web.archive.org/web/20181003060731/https://blog.legalsolutions.thomsonreuters.com
/legal-research/westlawnext-qa-session-refining-your-results] (responding to the question,
“[h]ow is relevancy determined?” by stating that “[r]elevancy is determined by a number of
factors that include search terms, KeyCite, key numbers, and usage by research
professionals”), with WestSearch: The World’s Most Advanced Legal Search Engine,
WESTLAW, http://info.legalsolutions.thomsonreuters.com/pdf/wln2/l-355700_v2.pdf
[https://perma.cc/26AR-FBTA] (stating only that Westlaw’s search algorithm emulates “the
best practices of experienced legal researchers”); and Westlaw Edge: The Most Intelligent
Legal Research Service Ever, THOMPSON REUTERS: LEGAL,
https://legal.thomsonreuters.com/en/products/westlaw [https://perma.cc/TV77-XSXT]
(making no mention of user behavior as a factor in the search process).
68
WestlawNext Q&A Session: Refining Your Results, supra note 67.
69
Id.
70
Matt Enis, Academic Ebook Sales Flat, Preference for E-Reference Up, LIBR. J. (Sept. 28,
2020), https://www.libraryjournal.com/?detailStory=academic-ebook-sales-flat-preference-
for-e-reference-up# [https://perma.cc/2H9S-QLJ2]; Bethany Cartwright, We Asked Our
Audience What They Really Think of PDF Ebooks: A HubSpot Experiment, HUBSPOT:
MARKETING (June 23, 2017, 8:00 AM), https://blog.hubspot.com/marketing/pdf-preferences-
184
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
more likely to print, email, or download a longer piece for later because
reading will require setting aside time and may need to be done over
multiple sessions. As with the time on page theory, we cannot be sure if
Westlaw, or other databases, still factor in print, email, and downloads.
If they do, it could lead to long articles appearing earlier in search
results, increasing their chances of being cited.
Ultimately, we cannot definitively determine why longer articles tend
to get more citations. Investing additional effort in researching theories,
like a feeling that longer articles are more authoritative, could provide
more insight. However, the proprietary nature of the algorithms that
deliver users’ results and likely play a role in citations (by putting some
articles earlier in search results) would make exploring some of our
other ideas challenging. So, while verifying these theories is outside the
scope of this paper, we thought readers would still find them interesting
and hope that we, or other researchers, can explore these sometimes-
competing ideas in the future.
Suggestions to Lengthen Articles
Having established a link between article length and number of
citations, we now offer suggestions on how to lengthen articles without
negatively impacting quality. Practically speaking, it is helpful to know
how many words are on a typical law journal page, as it might differ
from the word processor you use to draft your work. While it will vary
depending on the law journal, the general rule that law journals have
experiment [https://perma.cc/7D6L-N45F]; But see Matt Enis, College Students Prefer Print
for Long-Form Reading, Ebooks for Research, LIBR. J. (Mar. 27, 2018),
https://www.libraryjournal.com/?detailStory=college-students-prefer-print-long-form-reading-
ebooks-research-lj-survey [https://perma.cc/B9WF-KAEY] (indicating students prefer ebooks
when researching).
2021]
WILLEY & KNAPP
185
adopted is 5000 words equals ten law review pages.
71
To flesh that out
further:
Table 2: Conversion: Number of Words to Number of Pages
72
Number of Words
Number of Pages
5
,
000 words
10 pages
10,000 words
20 pages
15,000 words
30 pages
20,000 words
40 pages
25,000 words
50 pages
30
,000 words
60 pages
35,000 words
70 pages
40,000 words
80 pages
Based on our findings, you should target between 25,000 to 35,000
words, the equivalent of roughly 50-70 law journal pages.
Another practical consideration: many journals limit article length.
73
In
fact, a good number claim to limit article length below 63 pages.
74
So,
71
See Allen Rostron & Nancy Levit, Information for Submitting Articles to Law Reviews &
Journals, SSRN, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1019029
[https://perma.cc/KDZ5-KZL2].
72
Conversion chart created from understanding of Joint Statement of eleven law reviews
concerning article length. See, e.g., Article Length, DUKE L.J. (2021),
https://dlj.law.duke.edu/about/submissions/article-length/ [https://perma.cc/4B6H-Z9BN].
73
For example, Drexel Law Review, Florida Law Review, and North Carolina Law Review all
“prefer” submissions no longer than fifty law review pages (including footnotes); Harvard
says it “[w]ill not publish articles exceeding 30,000 words (roughly 60 law review pages)
except in extraordinary circumstances;” Stanford Law Review has a limit of 30,000 words
(roughly 60 law review pages) and “values brevity and looks favorably on pieces significantly
below the 30,000 word ceiling”; St. Thomas Law Journal “wants only 5,000 to 15,000 words
(roughly 30 law review pages), excluding footnotes. Rostron & Levit, supra note 71.
74
Id. (A few exceptions from the journals examined in Information for Submitting Articles to
Law Reviews & Journals, include Texas Law Review, where most articles should be within
the 40–70-page range, and Vanderbilt, which prefers “submissions of 20,000 to 35,000 words,
including text and footnotes (40 to 70 journal pages”)).
186
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
what should an author do when debating between hitting a word count
that could increase citations and one that will meet a journal’s
requirements? First, there is evidence that journals do not follow their
own article length limits, assuming that is the case, then authors have
no reason to limit length.
75
Second, if authors are faced with an actual
choice, they should consider the journals they are targeting.
76
If all are
similarly ranked, then our data suggest preferring journals that do not
have page limits.
Of course, when attempting to increase your word count, you should not
simply fill your article with extraneous words. Instead, focus on how
you can improve the article with additional length. The SEO world has
tools that can help with this by finding related topics and concepts.
77
However, SEO tools target and tend to work best with more general
topics and struggle to provide useful information on the complex,
nuanced, and niche topics of most law review articles.
78
So, while SEO
tools may be worth a look to see if they prompt you to think of a new
avenue, their actual results will likely be of limited use. Instead, focus
on applying the concepts behind the tools. Put yourself in the reader’s
75
Stephen M. Bainbridge, Law Review Word Limits Go Unenforced .... at Least at Harvard
and Yale, PROFESSORBAINBRIDGE.COM (Oct. 15, 2013),
https://www.professorbainbridge.com/professorbainbridgecom/2013/10/law-review-word-
limits-go-unenforced-at-least-at-harvard-and-yale.html [https://perma.cc/8ETV-6BM5].
76
As an aside, many of the journals with limits say something similar to the Cornell’s flagship
journal, “[t]he Cornell Law Review will not publish pieces exceeding 35,000 words except in
extraordinary circumstances.” Submitting Articles and Essays to Cornell Law Review,
CORNELL L. REV.: SUBMISSIONS, https://www.cornelllawreview.org/submissions/
[https://perma.cc/65D4-TA2V]. However, we did not see any journals that listed what
constitutes “extraordinary circumstances.” If it is related to the author’s name recognition,
then that would seem unfair because it is limiting the ability of those who most need citations
to get them, while allowing well-known authors to further pad their citation counts and
increase the gap over their less well-known peers. Rostron & Levit, supra note 71.
77
Joshua Hardwick, 10 Free Keyword Research Tools (That Aren’t Google Keyword
Planner), AHREFS BLOG (Apr. 7, 2021), https://ahrefs.com/blog/free-keyword-research-tools/
[https://perma.cc/D8TQ-HAEQ].
78
E.g., ANSWERTHEPUBLIC, https://answerthepublic.com/ (search run May 26, 2020) (a search
for “Agency Statutory Interpretation” pulls up zero questions, while a search for “fashion”
generates eighty).
2021]
WILLEY & KNAPP
187
shoes. What additional questions might they have that are relevant to
your paper? Consider running your article by colleagues or students to
see where they have questions or would like more detail. While this
manual approach lacks the amount of data behind SEO tools, it should
generate additional ideas. Do not focus solely on yourself; remember
your audience and keep in mind what will interest them.
Additionally, consider how extra length can help your article to rank
better in search algorithms.
79
If you can do it naturally, work in
synonyms and similar words for your most important points. Like SEO
experts, we only recommend this approach if you can do it naturally.
80
Not only will you hurt your chances of being cited if your writing is
poor, but Google eventually altered its algorithm to punish websites that
unnaturally crammed in keywords in an effort to rank higher.
81
We
suspect that legal search engines would do the same if the practice
became widespread in legal scholarship.
Example Articles
For those who are curious, we decided to include a few example articles
that support our findings. Remarkably, the most cited article in our
study, Big Data's Disparate Impact, came in at 62 pages (recall that the
average number of pages for articles in our top-articles sample is 63
pages) and, when the data was collected, had 177 citations. Was this
article by Cass Sunstein, the most cited author on Hein (28,239 citations
when the top-articles data was gathered)? No. It was authored by Solon
Barocas (cited 329 times when the data was gathered) and Andrew D.
Selbst (cited 266 times). On the other side, Government-Operated
79
Michelle Ebbs, 10 Easy Ways to Increase Your Citation Count: A Checklist, AJE SCHOLAR,
https://www.aje.com/arc/10-easy-ways-increase-your-citation-count-checklist/
[https://perma.cc/VU5A-K6XP]; e.g., Joran Beel et al., Academic Search Engine Optimization
(ASEO): Optimizing Scholarly Literature for Google Scholar & Co., 41 J. SCHOLARLY
PUBLISHING 176, 184 (2010).
80
How to Keep Keyword Density Natural and Avoid Keyword Stuffing, supra note 64.
81
See Irrelevant Keywords, GOOGLE SEARCH CENTRAL,
https://support.google.com/webmasters/answer/66358?hl=en [https://perma.cc/GA9U-JEX2]
(stating that overloading with keywords “can harm your site's ranking”).
188
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Drones and Data Retention, comes in at 22 pages and has only 1
citation, despite the timely and compelling topic. Of course, these are
just examples used to illustrate the point. Our data set also has counter
examples, like Cass Sunstein’s Chevron as Law, which had only 3
citations when the data was gathered despite the recognizable author
and its 72-page length.
Are Longer Articles Really Better?
Our article focuses on what types of articles are more likely to be cited.
While the data indicates longer articles are more likely to be cited, this
does not mean that they are actually better. Personally, we dread the
idea of reading a 60+ page law review article and agree with Scott
Dodson that shorter is better.
82
However, with longer articles being cited
more often, it seems likely they are here to stay. Changing this trend
would require putting less emphasis on citations and finding another
proxy for value.
Characteristic Two: Short Titles Without Colons
Title Length
The average title in the top-articles sample had 52 characters (including
spaces). The bottom-articles, author over 100 citations sample
averaged 70 title characters, while the bottom-articles, 3 citations
sample had an average of 73 characters per title. Within the top-articles
sample, the two most frequently occurring title lengths were 27 and 32
characters. In the top-articles sample, 60 percent of the article titles had
between 19-57 characters, while 66 percent of the titles in the bottom-
articles, author over 100 citations sample fell between 34-98
characters, and 64 percent of the bottom-articles, 3 citations sample
titles were between 25-89 characters (see chart below for a visual
overview of the distribution). Further, 76 percent of articles in the top-
82
Dodson, supra note 58, at 667.
2021]
WILLEY & KNAPP
189
articles sample fell below 70 characters and only 7 percent were over
100 characters, while in the bottom-articles, author over 100 citations
sample, 54 percent fell under 70 characters, and 18 percent came in over
100 characters. In sum, while “short” is a relative term, we found that
articles with more citations tend to have shorter titles when compared
to less cited articles.
While character length is the most precise measure of title length, word
count provides a different and useful perspective. In the top-articles
sample, titles averaged 7.30 words, while the bottom-articles, author
over 100 citations sample averaged 10.28 words, and the bottom-
articles, 3 citations sample averaged a nearly identical 10.67 words. 60
percent of the titles in the top-articles sample had between 1 and 7
words and 84 percent fell below 12 words, with the two most common
title lengths coming in at 3 and 5 words respectively. Thus, titles with
fewer words tended to be cited more often in our dataset.
Not all articles followed the trends in their respective samples. The
longest title in the top-articles sample came in at 210 characters, while
the shortest title was 5 characters. Titles in the bottom-articles, author
over 100 citations sample ranged from 7 to 194 characters, with the
bottom-articles, 3 citations sample titles coming in between 11 and
218 characters. As with article length, the ranges indicate that we found
title length trends, not rules.
Title Colons
Whether a title had a colon also proved statistically significant, with
more cited articles less likely to include a colon in the title. Thirty-two
percent of titles in the top-articles sample had colons, while 55 percent
of titles in the bottom-articles, author over 100 citations sample, and
50 percent of titles in bottom-articles, 3 citations sample had colons.
190
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Additional Academic Studies
Studies outside the legal realm indicate that whether long or short titles
are preferred varies by discipline. Although it did not provide an optimal
length, a study of 140,000 science, technology, medicine, social
sciences, and arts and humanities articles found that when reviewing the
data set as a whole, shorter titles tend to receive more citations than
longer ones.
83
However, this did not hold across all journals studied,
with longer titles performing better in some journals.
84
Further, another
study found that shorter titles received more citations in Sociology and
Applied Physics, but in Internal & General Medicine longer titles did
better.
85
The same study also looked at colons and found that colons in
the title lead to more citations in Applied Physics but fewer citations in
General and Internal Medicine.
86
Within the legal arena, the findings of Ayers and Vars, also cited in the
previous section, align with ours for title length but not the presence of
a colon.
87
In their study of three top law reviews, Ayers and Vars found
that “articles with shorter titles received significantly more citations
than articles with longer titles.”
88
They also found that “having a colon
in the title significantly reduced the probability of falling into the
bottom 10 percent.”
89
We do not have Ayers and Vars data, so we can
only speculate on why we came to different conclusions on the presence
83
Adrian Letchford, Helen Susannah Moat & Tobias Preis, The Advantage of Short Paper
Titles, 2 ROYAL SOC’Y OPEN SCI. 1, 5 (2015).
84
Id. (chart indicating some journals, like Lancet, have more citations if a title is longer).
85
van Wesel et al., supra note 1, at 1606.
86
Id.
87
T. Liskiewicz et al., Factors Affecting the Citations of Papers in Tribology Journals, 126
SCIENTOMETRICS 3321, 3322 (2021).
88
Ayres & Vars, supra note 21, at 440.
89
See id. at 443 (cautioning that “[t]he least-cited regression results are subject to all of these
interpretations plus the possibility that the results relate to variance in citation rates. For
example, the finding that having a colon significantly reduced the likelihood of an article
falling into the least-cited decile might indicate merely that such articles have a lower variance
in citation.”).
2021]
WILLEY & KNAPP
191
of a colon. One possibility is that none of the three subsets we pulled
came from the bottom 10 percent of the set of over 200,000 articles that
we gathered our subsets from and that is why our findings differ.
Perhaps a comparison of the bottom 10 percent would align with Ayers
and Vars. However, we want to help authors become well-cited, not
merely avoid the bottom 10 percent, which in our data set would include
only articles with 0 citations. So, although the lack of alignment with
Ayers and Vars weakens our confidence, we still recommend avoiding
a colon in the title if possible. The chart below shows the distribution of
articles in the data sets.
192
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Figure 3: Title Character Count
Figure 4: Colon in Title
0
20
40
60
80
100
Avg. Character Count % under 70 characters % over 100 characters
Title Characters
Top Bottom "100" Bottom "3"
32%
55%
50%
0%
10%
20%
30%
40%
50%
60%
Top Bottom "100" Bottom "3"
% of Titles with Colon
2021]
WILLEY & KNAPP
193
Search Engines and SEO Experts Also Favor Shorter Titles
There are a few differences to consider when comparing titles on the
web to those in law reviews. First, on the web, “title” can have a few
meanings. It can be the heading that appears at the top of a webpage that
is coded with an HTML h1 or h2 tag. Title could also mean the meta
title tag (i.e., <title>Example Title</title>) that Google usually displays
in search results.
90
In practice, the page title and title tag are usually the
same or very similar, but they do not have to be (see screenshots below
illustrating the title tag and page title). Because the title tag displays in
search results and influences whether users will click into a page from
search results, the title tag is probably the more discussed “title” in SEO
circles. Below we will specify which “title” we are referring to
whenever possible.
90
See Title Tag, MOZ: FUNDAMENTALS, https://moz.com/learn/seo/title-tag
[https://perma.cc/2EDX-PC5C] (explaining that Google will not always use your title tag in
search results); Michal Ugor, We Analyzed 1.5 Million Title Tags to Find Out What the Ideal
Length Is for SEO in 2020 (Hint: It’s Not 80 Characters Anymore), AUTHORITYHACKER (Sept.
20, 2019), https://www.authorityhacker.com/seo-title-tags/ [https://perma.cc/6STB-GGXM]
[hereinafter Ugor, We Analyzed 1.5 Million Title Tags] (finding that Google often makes
minor edits to title tags and uses an h tag when no title tag is available, while completely
ignoring their own advice to limit title length).
194
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Figure 5: Title Tag
Figure 6: Page Title
Second, Google typically only displays the first 50-60 characters of a
meta title tag in search results (screenshot below shows a cutoff title).
91
91
Google allows the title to take up to 600 pixels (px). The actual characters that fit in 600px
can vary based on the characters themselves—an “i" takes less space than a “w.”
Consequently, SEO experts disagree on exactly how many characters can fit in search results.
Compare Title Tag, supra note 90 (claiming 50-60 characters), with Ugor, We Analyzed 1.5
Million Title Tags, supra note 90 (claiming 80 characters is the max but adding that Google
often adds in a brand, trimming available space closer to 50 characters).
2021]
WILLEY & KNAPP
195
As discussed below, this limit on search result title lengths may hurt
longer titles more than it would in legal databases, where titles are rarely
cut off.
92
Further, Google does recommend short titles.
93
Savvy website
owners are aware of this and of Google’s length limit and strive to make
sure that the key portion of their title is shown in search results, often
shortening titles to keep them under Google’s limit.
94
Figure 7: Title Cutoff
With that background out of the way, we can now examine whether the
web also favors shorter titles. The webpage studies we looked at
presented results surprisingly similar to what we found in our study of
law review articles. When it comes to search results, a study of 1.5
million meta title tags found that shorter titles perform better, with the
top performers having titles between 45-50 characters, close to the 52-
92
For example, Westlaw, HeinOnline, and LexisNexis do not cutoff titles in search results.
93
Create Good Titles and Snippets in Search Results, SEARCH CONSOLE HELP CENTER,
https://support.google.com/webmasters/answer/35624?hl=en [https://perma.cc/YR8Y-
BGW2].
94
Compare Title Tag, supra note 90.
196
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
character average of the articles in our top-articles sample.
95
Backlinko
found people are more likely to click search results with meta title tags
of 15-40 characters.
96
Another well-documented study by Etsy also
found shorter meta title tags performed better, although they did not
provide an optimal length.
97
Finally, a study that looked at number of
words, instead of characters, found that titles with fewer words tend to
perform better, with 8 words being optimal, just 1 word more than the
average in our top-articles sample.
98
These studies indicate that Google’s algorithm, and presumably the
users Google’s algorithm is designed to serve, prefer shorter titles. Still,
because of Google’s title truncation, most of the “longer” titles that
these SEO studies looked at were around 60 characters.
99
Therefore,
these studies do not show whether users would prefer titles of over 60
characters.
Social media platforms tend to allow longer headlines, so examining
them moves us away from the restraints Google puts on title length.
100
95
Ugor, We Analyzed 1.5 Million Title Tags, supra note 90. When reviewing this source, keep
in mind that because Google cuts off titles around 60 characters, the titles in this study max
out at roughly 60 characters. However, this study only looked at the character length of titles
that displayed in search results, not the actual length of the meta title tag. So, some of the
actual titles in this study could be significantly longer but were truncated by Google.
96
Brian Dean, We Analyzed 5 Million Google Search Results: Here’s What We Learned About
Organic Click Through Rate, BACKLINKO (Aug. 27, 2019), https://backlinko.com/google-ctr-
stats [https://perma.cc/2V54-BR89].
97
Bill Ulammandakh, SEO Title Tag Optimization at Etsy: Experimental Design and Causal
Inference, ETSY: CODE AS CRAFT (Oct. 25, 2016), https://codeascraft.com/2016/10/25/seo-title-
tag-optimization/ [https://perma.cc/UVA7-VG7J].
98
Matthew Barby, How to Rank Number One in Google: A Study of
1 Million Pages, MATTHEWBARBY (Jan. 16, 2020), https://www.matthewbarby.com/ranking-
in-google/ [https://perma.cc/G48P-37QZ].
99
Ugor, We Analyzed 1.5 Million Title Tags, supra note 90.
100
Confusingly, when webpages are shared on social media, the “title” can be the title tag, as
with search results or it can be another tag specifically designed only to show on social
2021]
WILLEY & KNAPP
197
Studies that looked at headline length’s impact on social media shares
(i.e., which webpages were more likely to be shared on social media),
found that longer headlines outperformed shorter ones.
101
This may be
due to how social media posts often display website headlines, with the
headline taking up a large portion (sometimes all) of the post, making
more descriptive headlines especially useful. Another possibility may
have to do with people’s purpose for visiting social media versus
searching Google. People often visit social media for leisure. LinkedIn
is an exception to this as people generally visit for a career-related
purpose. A study found that the optimal headline length for LinkedIn
shares (around 8-10 words) is significantly lower than the optimal
length for Facebook shares.
102
Perhaps focused users with work on their
minds have different preferences than users browsing in their free time.
Regardless of the reasons longer titles do better on social media,
webpages are more like law review articles than social media posts.
Therefore, we think that the better performance of shorter title tags in
search results and Google’s stated preference for shorter titles support
our findings and recommendation that law review authors write shorter
titles.
media. Michal Ugor, How to Write the Best Headlines for Shares, Links & Traffic [Data
Powered Guide], AUTHORITYHACKER (June 25, 2019),
https://www.authorityhacker.com/headlines/ [https://perma.cc/RQN4-E54D] [hereinafter
Ugor, How to Write the Best Headlines].
101
See BuzzSumo Research: 100 Mil Headlines Analysis. Here’s What We
Learned, BUZZSUMO: BLOG (June 26, 2017), https://buzzsumo.com/blog/most-shared-
headlines-study/#words [https://perma.cc/XP2N-B2ZX] (finding 80-95 characters the optimal
length for sharing titles on Facebook); see also, Ugor, How to Write the Best Headlines, supra
note 100 (finding that the average length of the top 1 percent of titles was 70 characters, while
the average length of all titles was 65 characters).
102
BuzzSumo Research: 100 Mil Headlines Analysis. Here’s What We Learned, supra note
101.
198
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
No SEO Studies on Use of Colons
Our research did not uncover any SEO data on colons in titles.
103
However, Orin Kerr recommends avoiding colons, saying the format “is
popular among student notes” and should be avoided.
104
Hein’s
ScholarRank top 250 authors ranked Kerr at number five as of this
writing. So, although he does not provide any data, it seems he may
know what he is talking about.
105
Why Are Shorter Titles Without Colons Preferred?
Citing authors may prefer shorter titles because they can be read and
understood faster. Or, perhaps, a lengthy, hard to understand title, leads
people to assume that the article itself will be hard to understand.
Additionally, some legal scholars likely use Google, either out of
preference or necessity. This means that Google’s limit on characters
may play a role here: a truncated title may not make sense, leading the
searcher to skip it and move to the next one. Take Presidential
Administration: How Implementing Unitary Executive Theory Can
Undermine Accountability (a title from one of our lower samples),
Google truncates this result (screenshot below), leaving the reader
wondering, “unitary” what?
103
The SEO discussions of colons in titles generally focus on how Google interprets them —
as colons if you are curious. Cara Bowles, Title Tag SEO: 50 Questions Answered,
NORTHCUTT, https://northcutt.com/seo/53-enlightening-questions-about-title-
tags/ [https://perma.cc/4DMT-QUV6].
104
Orin Kerr, Tips on Placing Law Review Articles, PRAWFSBLAWG (Oct. 1, 2014, 12:24
AM), https://prawfsblawg.blogs.com/prawfsblawg/2014/10/tips-on-placing-law-review-
articles.html [https://perma.cc/TL25-M5TC].
105
ScholarRank’s Top 250 Authors in HeinOnline, HEINONLINE,
https://home.heinonline.org/top_authors/ [https://perma.cc/AZG8-EVY8].
2021]
WILLEY & KNAPP
199
Figure 8: Title Truncation
As for colons, we suspect that a lower occurrence of colons in titles is
at least partly related to title length. Colons are typically used in titles
to explain something, making it more likely that a title with a colon will
need to be longer. Further, as demonstrated in the Presidential
Administration title, adding a colon essentially allows authors to include
two titles: a short one and a longer explanatory one.
Example Titles
We think that the most-cited article in our study did an excellent job
succinctly explaining what the article is about: Big Data's Disparate
Impact. Scanning through articles in the top-articles sample, it appears
many authors simply put one or two sets of descriptive words on their
topic: Information Fiduciaries and the First Amendment; Unorthodox
Lawmaking, Unorthodox Rulemaking; Unequal Protection; and
Sensitive Information. Some titles in the less cited samples take a similar
approach, but tack on an explanatory phrase: Presidential
Administration: How Implementing Unitary Executive Theory Can
Undermine Accountability; Indigenous Identity, Cultural Harm, and the
Politics of Cultural Production: A Commentary on Riley and
Carpenter's Owning Red; Community Land Trusts: Why Now Is the
Time to Integrate This Housing Activists' Tool into Local Government
Affordable Housing Policies; The Elephant in the Room in Debates
about Universal Jurisdiction: Diasporas, Duties of Hospitality, and the
Constitution of the Political; and Warrantless Blood Tests, Drunk
Driving, and Exigent Circumstances: Preserving the Liberty Guarantee
of the Fourth Amendments While Evolving the Exceptions to the
200
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Warrant Requirements. Could it be that not providing a full description
in the title prompts users to click a result and read it?
In sum, we recommend limiting title lengths, using the data we provided
in this section as a guideline. Avoiding colons in titles should also
increase your chances of getting cited.
Characteristic Three: Write on A Popular Topic
106
Writing on a popular topic to increase citations is a more obvious idea
than the previously discussed items. Still, seeing numbers on this
characteristic should drive home its importance. As we have with the
other characteristics, we lead with numbers from our study.
HeinOnline uses artificial intelligence (AI) to assign one or more topics
to each article.
107
Reviewing those topics revealed some interesting
trends. Twelve percent of the articles in the top-articles sample
included “technology” or “science and technology” as a topic,
compared to 5 percent in the bottom-articles, author over 100
citations sample, and 7 percent in the bottom-articles, 3 citations
sample. “Immigration” came up as a topic in 6 percent of articles from
the top-articles sample; 1 percent of articles from the bottom-articles,
author over 100 citations sample; and 2 percent of articles from the
bottom-articles, 3 citations sample. “Race” occurred in 7 percent of
top-articles; 2 percent of bottom-articles, author over 100 citations;
and 5 percent of bottom-articles, 3 citations articles. The higher
percentages in the bottom-articles, 3 citations sample when compared
to the bottom-articles, author over 100 citations sample may be due
to the significant number of student notes in the former sample. In our
yearly meetings with journal students, we have observed that they tend
106
We did not run an analysis to determine if the items in this section are statistically
significant.
107
Hein Online’s Suite of Tools, HEINONLINE, https://home.heinonline.org/tools/
[https://perma.cc/3MNY-QTVS].
2021]
WILLEY & KNAPP
201
to gravitate toward trending, popular topics. We suspect this is true at
most law schools. Yet, student pieces are less likely to be cited,
108
presumably, even if focused on a trending topic. This may explain the
higher percentages in the bottom-articles, 3 citations sample.
Moving to topics some might view as less “popular,” we found the gap
smaller when it came to articles with “arbitration” listed as a topic. Here,
2.39 percent of top-articles articles had “arbitration,” compared with
1.41 percent in the bottom-articles, author over 100 citations and 1.53
percent in the bottom-articles, 3 citations samples. We also found the
numbers closer with “comparative law” as the subject, where 3 percent
of articles in the top-articles sample had “comparative law” as a topic,
compared with 5 percent of bottom-articles, author over 100
citations, and 4 percent of bottom-articles, 3 citations.
These comparisons intrigued us, but further breaking down our samples
by topic led to a limited number of articles per topic, making us less
confident in our conclusions. We wanted to expand the sample size to
confirm what we were seeing. So, we decided to look at topics within
the entire set of 242,924 articles. To do that, we pulled up all the articles
classified with a topic and calculated the number of citations per article
across certain topics. Articles with “technology” as a topic came in at
0.65 citations per article, while the less headline-grabbing “arbitration”
had 0.49 citations per article. These results matched our assumption that
trending topics would get cited more often. Then, we dug further and
found that “comparative law” gets 0.72 citations per article, more than
the presumably attention-grabbing topic of technology. Also, we
expected “race” would be a well-cited topic, and it was with 1.46
citations per article. However, the size of race’s gap over technology
was larger than we had anticipated. We decided to examine the broad
topic of technology a little closer by seeing if more trendy niche topics
108
Harrison & Mashburn, supra note 55, at 65 (finding that student works averaged 8.2
citations per article compared to 35.5 for non-student articles).
202
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
within technology led to different numbers. They did: the topic of
“technology” with the search term “bitcoin” came in at 1.63 citations
per article, while “technology” and “robot” garnered 2.08. Additionally,
Hein has a separate “artificial intelligence” topic. It had 1.25 citations
per article. Overall, these numbers indicate a connection between the
topics you choose to write on and your chances of being cited. The
breakdown of “technology” shows that picking a broadly popular
subject is not sufficient; you need to home in on a narrow and popular
portion of that broader subject.
Although they looked at different subjects, Ayres and Frederick’s study
of citations in law reviews also observed that articles on certain subjects
tend to garner more citations than others.
109
In their study, articles on
jurisprudence, feminism, and Critical Legal Studies tended to get more
citations, while those on international law and criminal law got less.
110
Ayres and Frederick conducted their study in 2000. Because tastes
change, their data may have less relevance today for authors looking to
write on popular topics. Regardless, their findings support our larger
claim: topic selection can have an impact on potential citations.
111
SEO and Topic Selection
Not surprisingly, SEO experts have also found a correlation between
topic and webpage popularity.
112
As an example, HubSpot did a study
that, in part, looked at their blog posts to see how topic selection
influenced popularity.
113
They found that certain topics were cited more
often than others.
114
However, there are comparatively few SEO studies
109
Ayres & Vars, supra note 21, at 437-39.
110
Id.
111
Id.
112
Matthew Howells-Barby, The Anatomy of a Shareable, Linkable & Popular Post: A Study
of Our Marketing Blog, HUBSPOT (July 28, 2017), https://blog.hubspot.com/marketing/seo-
social-media-study [https://perma.cc/8BXU-LYJN].
113
Id.
114
Id.
2021]
WILLEY & KNAPP
203
on topic selection. This is almost certainly because the proliferation of
tools designed to analyze topics makes it easy for users to quickly run
their own, real-time studies on various topics, as we detail later in this
section.
Incorporating Popular Topics
Depending on your area of expertise, incorporating a popular topic may
require some creativity. But with some effort, it can usually be done.
Andrew Ferguson provides an example of this by weaving tech-related
topics with his expertise in criminal law.
115
Ferguson authored the
twelfth most cited article in our data set: Big Data and Predictive
Reasonable Suspicion.
116
The article incorporates a popular topic (big
data) with an area of the law Ferguson knows well. Looking at the titles
in Ferguson’s Hein Author Profile reveals use of this approach several
times before: The Internet of Things and the Fourth Amendment of
Effects
117
and The Smart Fourth Amendment.
118
Another example of this
would be intellectual property (IP) experts writing on ownership of
works created by artificial intelligence systems.
119
While this approach will not work with all trending topics, the concept
is straightforward. First, find a popular topic that interests you. Then,
consider how your area of expertise might be applied to the topic. If you
get stuck, move on to another topic. The next section aims to help with
115
Andrew Guthrie Ferguson, AM. U.: W. COLL. L.,
https://www.wcl.american.edu/community/faculty/profile/aferguson/publications [https://perm
a.cc/9LSD-4HZ6]. Our observations here are based simply on Ferguson’s CV and the data
from Hein. We cannot say whether Ferguson consciously planned this strategy.
116
See Andrew Ferguson, Big Data and Predictive Reasonable Suspicion, 163 U. PA. L.
REV. 327 (2015).
117
See Andrew Ferguson, The Internet of Things and the Fourth Amendment of Effects, 104
CALIF. L. REV. 805 (2016).
118
See Andrew Ferguson, The “Smart” Fourth Amendment, 102 CORNELL L. REV. 547 (2016-
2017).
119
E.g., Shlomit Yanisky-Ravid, Generating Rembrandt: Artificial Intelligence, Copyright,
and Accountability in the 3A Era, 2017 MICH. ST. L. REV. 659 (2017).
204
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
the first step by introducing SEO tools designed to gauge and compare
topic popularity.
Finding Popular Topics
The tools used by web authors were designed for the web.
Consequently, they have limitations in the academic world. Below we
touch on a few tools, explaining their potential uses and pitfalls.
Answer the Public generates content ideas based on data it scrapes from
Google’s auto-populated suggestions.
120
Depending on the topic, it may
have some uses for scholars. Entering “privacy laws” pulled up a few
interesting suggestions: “what privacy laws apply to the operation of a
drone,” “how privacy laws affect business operations,” “privacy vs law
enforcement,” and, the surprisingly specific, “how privacy laws can
impact the school nurse.”
121
However, we found many topics too narrow
for the tool. For example, a search of answerthepublic.com for
“Comparative law” pulls up 23 questions, eight comparisons, and eight
related topics that the tool has found searchers are asking.
122
Unfortunately, the questions, comparisons, and topics are too basic for
a scholarly article.
123
A few illustrative examples, include “What is
comparative law?” and “Why is comparative law important?” Also, we
should note that Answer the Public seeks to generate ideas for writing
on topics, not rate a topic’s popularity.
124
120
Sophie Coley, How to Use AnswerThePublic, SEARCHLISTENING (Sept. 3, 2019),
https://searchlistening.com/how-to-use-answerthepublic-com-become-an-expert-with-search-
listening/ [https://perma.cc/3V23-86BA] (watch video).
121
Search Results from “Privacy Laws”, ANSWERTHEPUBLIC,
https://answerthepublic.com/ (search run June 30, 2020).
122
Search Results from “Comparative Law”, ANSWERTHEPUBLIC,
https://answerthepublic.com/ (search run May 26, 2020).
123
Id.
124
ANSWER PUBLIC, https://answerthepublic.com/ [https://perma.cc/FY84-7QQU].
2021]
WILLEY & KNAPP
205
Moz’s Keyword Explorer “help[s] you figure out what keywords people
are searching, what keywords you’re already ranking for, the demand
for certain searches, and the strength of other sites competing for your
target keywords.”
125
It does a little better on “comparative law” than
Answer the Public, offering expected monthly volume (a rough
approximation of popularity) plus a few related keywords (many of
which Moz has no data on).
126
However, the issue academic scholars
may have with Moz, and similar SEO tools, is their focus on existing
websites. Moz rates existing websites that cover your topic and how
hard it will be for you to rank higher than them.
127
This is of little use
to academic authors who are not interested in ranking against websites.
Further, Keyword Explorer focuses on a topic’s popularity across
established websites, not within society as a whole.
128
This will make it
less helpful the more recently popular a topic is.
As the name suggests, Google Trends focuses on identifying trending
topics in its widely used search engine.
129
Users can go to the homepage
and get an overview of the most popular topics on Google.
130
While
many of these topics may be challenging to turn into legal articles
(“national donut day”), others will not (“third amendment”).
131
Further,
125
Keyword Explorer Overview, MOZ, https://moz.com/help/keyword-explorer/getting-
started/overview [https://perma.cc/C6MF-GR3W].
126
Search Results from “Comparative Law”, MOZ, https://moz.com/explorer (search run June
3, 2020).
127
Id.
128
Id.
129
FAQ About Google Trends Data, GOOGLE HELP,
https://support.google.com/trends/answer/4365533?hl=en [https://perma.cc/HS99-7AKW].
130
Explore What the World is Searching, GOOGLE TRENDS,
https://trends.google.com/trends/?geo=US [https://perma.cc/LR2T-2NDJ].
131
Trending topics on June 5, 2020 and October 2, 2020, GOOGLE
TRENDS, https://trends.google.com/trends/?geo=US (“national donut day” and “third
amendment” were listed as trending topics on June, 5th, 2020. A few months later, on October
2nd, “Trump Covid” and “Broncos vs Jets”).
206
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
users can enter and compare their own topics. A search for “comparative
law” shows a varied trendline with interest peaking in late 2018.
132
Figure 9: Google Trends, “Comparative Law” Example
Since this trendline is so varied, with no clear increasing trend,
“comparative law” does not look like a timely topic. This becomes more
apparent when we compare it to “bitcoin” (graph below). The results of
the comparison show that bitcoin (red line) is generating significantly
more interest than comparative law (flat blue line) and that the optimal
moment to write about bitcoin may have passed (red line spike in
2017).
133
This chart also indicates that, at least in this case, popularity
in search tracked with what we found in our review of topics on Hein:
bitcoin was the more searched for term on Google, and articles on
bitcoin also garnered more citations per article than comparative law.
132
Search Results for “Comparative Law”, GOOGLE TRENDS,
https://trends.google.com/trends/?geo=US (search run June 3, 2020 for past 5 years).
133
Search Results for “Comparative Law” and “Bitcoin”, GOOGLE TRENDS,
https://trends.google.com/trends/?geo=US (search run June 3, 2020 for past 5 years).
2021]
WILLEY & KNAPP
207
Figure 10: Google Trends, “bitcoin” example
This comparison brings up an issue academic users should consider
when working with Google Trends. It performs well for comparisons
where the terms have only a few logical meanings, like bitcoin and
comparative law. However, it can falter where terms have multiple
meanings. For example, when looking at “race,” the “related queries”
include items like “Alabama senate race” and “amazing race.”
134
So,
when using Google Trends, make sure to scroll down below the graph
to confirm Google has not lumped in unintended topics with its analysis
of your terms. As the discussed examples show, authors can use Google
Trends to gauge relative popularity and compare two topics they are
considering to determine which is more popular. Authors can also look
at a single topic to see if its popularity is rising or falling. For example,
a search for “Posse Comitatus Act” on June 3, 2020, shows a likely
spike in traffic (dotted line).
135
134
Search Results for “Race”, GOOGLE TRENDS, https://trends.google.com/trends/?geo=US
(search run June 5, 2020 for past 5 years).
135
Search Results for “Posse Comitatus Act”, GOOGLE TRENDS,
https://trends.google.com/trends/explore?date=2015-06-03%202020-06-
03&geo=US&q=%2Fm%2F0g_sg (search run June 3, 2020).
208
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Figure 11: Google Trends, “Posse Comitatus Act” Example
Using the tools discussed here and other SEO targeted tools can help
you in identifying the popularity of a topic or deciding between multiple
topics. Still, none of these tools are perfect, largely because they were
made with the web in mind. Therefore, it remains important to keep
yourself up-to-date and open to how seemingly unconnected trends can
tie into your scholarship. The best time to identify a trend is before it
has even started, something none of these tools claim to do.
Competitor Analysis
In the SEO world, web authors look for popular topics that are likely to
get plenty of traffic but have not been well-covered already.
136
This type
of check is often referred to as a “competitor analysis.”
137
Moz’s
Keyword Explorer, also discussed above, provides one example of a
tool designed to help web authors conduct a competitor analysis. It has
a “difficulty” metric, which “shows [web authors] how tough it would
be to rank in the top 10 spots for [a] keyword.”
138
Although SEO tools
will not allow legal authors to analyze competition from existing
scholarly articles, we recommend adopting a version of the SEO
competitor analysis during the pre-emption check. In addition to seeing
136
Joshua Hardwick, How to Do an SEO Competitor Analysis [Template Included], AHREFS
BLOG (May 14, 2020), https://ahrefs.com/blog/competitive-analysis/ [https://perma.cc/DAN6-
F2WR].
137
Id.
138
Difficulty, MOZ, https://moz.com/help/keyword-explorer/keyword-metrics/difficulty
[https://perma.cc/NK22-CH8E].
2021]
WILLEY & KNAPP
209
if anyone has covered your exact topic, also consider how likely it is
people will cite your work over that of an existing piece. Has a well-
known scholar already authored a popular article on a similar point to
yours? If so, you may want to consider a different topic. Many authors
likely do this already, but we want to suggest it explicitly. As with SEO,
we believe that the popular topics that have not been widely written on
represent the sweet spot for legal authors.
To summarize our recommendations in this section, start by identifying
topics that interest you. Examine the topics for popularity and consider
how you can incorporate them with your legal knowledge. Next,
analyze the competition to see how likely your article is to be cited over
existing articles. Ideally, you will find a topic that is popular but not
widely written on. Consider all these factors together when choosing a
topic.
Additional Items to Consider
This section covers some additional items we observed in our study or
came across in our SEO research. While important, we either did not
think they required the same depth of discussion we gave to the prior
items or did not have enough data to give full coverage.
Publish in a Widely Accessible Journal
A study on the impact ease of access has on citation counts in finance
and economics journals found that making journals freely accessible
increased citation counts in lower ranked journals.
139
Our data support
a similar conclusion for law articles. In this section, we look at an
embargoed journal (full text withheld online for a period of time) to
demonstrate.
139
Benedicte Millet-Reyes & Barrie A. Bailey, Internet Access, Journal Ranking, and
Citation Performance, 34 J. FIN. EDUC. 28, 39 (2008).
210
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Of the top ten journals in Washington & Lee’s Law Journal Rankings,
140
only one has an embargo period with HeinOnline.
141
The Supreme
Court Review’s embargo requires HeinOnline to wait three years before
sharing full text articles.
142
Out of the top-articles sample of 502
articles we reviewed, the Supreme Court Review came in last for
number of articles in the sample, with only one article or 0.2 percent of
the sample. The next closest was the Fordham Law Review with eight
articles or 1.6 percent, then the California Law Review with fifteen
articles or 3 percent. Of course, we need to point out that the Supreme
Court Review only publishes one issue per year, compared to Fordham
and California’s 6 issues.
143
Still, based solely on number of issues, the
Supreme Court review should receive 1/6
th
the citations of the other
journals. Instead, it gets 1/8
th
and 1/15
th
of the closest competitors and
only 1/39
th
of the most cited journal in our study, the Yale Law Journal,
which publishes 8 issues per year.
144
The Supreme Court Review’s low citation count is not due to a lack of
inclusion in Hein’s citation count because Hein includes citation
analytics for embargoed content.
145
To drill down a little further, we
140
We looked at the 2018 W&L Law Journal Rankings for our top ten, because “citations per
year peek at 4 years after publication.” Ayres & Vars, supra note 21, at 427. In 2015, when
most of the articles in the top sample were authored, the Supreme Court Review came in at
eighteenth, the California Law Review at seventeenth, and Fordham at nineteenth. In 2019,
the Supreme Court Review moved from tenth to ninth; the California Law Review went from
ninth to seventh, while Fordham dropped from eighth to twenty-first. W&L Law Journal
Rankings, WASH. & LEE UNIV. SCH. L.,
https://managementtools4.wlu.edu/LawJournals/Default.aspx [https://perma.cc/KE8B-4CCL]
(select the appropriate year; check combined score; then click submit).
141
HEINONLINE, https://heinonline.org/HOL/Welcome [https://perma.cc/W45G-RX7S] (login
to Hein; navigate to the Supreme Court Review’s page; and click the “i” next to the title).
142
Id.
143
Id. (login to Hein; navigate to each journal’s page; and click the “i” next to the title).
144
Id.
145
Does Hein’s ScholarCheck Include Embargoed Journals in the ‘Cited by’ Metrics?,
HEINONLINE: KNOWLEDGE BASE, https://help.heinonline.org/kb/u-s-news-is-considering-
using-heins-scholarcheck-to-evaluate-law-school-scholarly-impact/ [https://perma.cc/XL4L-
V2PH].
2021]
WILLEY & KNAPP
211
looked at Fordham Law Review articles (477) and Supreme Court
Review articles (49) over the period we pulled our data from (2015-
2019, inclusive). Over that entire time period, Fordham articles
averaged 4.17 citations per article, while Supreme Court Review articles
had 0.82 citations per article.
For the issues of the Supreme Court Review in our study sample, 2018
and 2019 were embargoed as of this writing. Full text is provided for
2015, 2016, and 2017. Table 3 compares the number of citations per
article (an average we calculated by counting articles and dividing by
the total number of citations to all of those articles; student notes in the
Fordham Law Review were excluded).
212
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Table 3: Average Citations Per Article for Embargoed Journal and
Full-Text Accessible Journals (mean, median, mode).
Supreme Court Review 2016
Supreme Court Review 2018
10.44
6
2
2.58
2
1
8.85
5
2
6.54
5
6
Fordham Law Review 2015-16
Fordham Law Review 2017-18
Of the 4 volumes shown here, only Supreme Court Review 2018 is
embargoed. Full text is available for the other three volumes.
A t-test indicates there is a significant difference between the mean for
the Supreme Court Review in 2016 compared to Supreme Court
Review in 2018. There is no significant difference between the
Fordham Law Review 2017-18 and Supreme Court Review 2018.
The data for this table was collected in February 2021.
The Supreme Court Review’s average citations per article are much
lower in a year where full text is not available. In conclusion, the harder
you make it for would be citing authors (and the search engines they
use) to access your material, the less likely they are to find and cite it.
Other things being equal, we recommend publishing in a widely
accessible journal.
Publish in a Top Journal
You probably already knew this one, but here are our data and thoughts.
The top-ten journals in Washington & Lee’s Law Journal Rankings for
2021]
WILLEY & KNAPP
213
2018 made up 37 percent of the articles in our top-articles sample, 4
percent of our bottom-articles, author over 100 citations sample, and
5 percent of our bottom-articles, 3 citations sample. In our top-articles
sample, the top 10 journals account for more than their share of the total
citations in the sample: 45 percent, or 8695, citations out of 19,516 total
citations. Additionally, Jeffrey Harrison and Amy Mashburn
specifically examined the impact that the rank of a journal’s school had
on citations and found that journals published at higher ranked schools
tend to get more scholarly citations.
146
So, with the caveats listed in this
article, publish in the best journal you can. Yale, Harvard, Stanford, and
Columbia’s flagship journals seem highly regarded across a range of
sources.
147
Outside of these, you will see some variation depending on
the ranking system. Harrison and Mashburn found that specialty
journals published at schools ranked well by U.S. News tended to
receive more citations than similar journals at lower ranked schools.
148
Ultimately, we recommend looking at multiple journal ranking metrics
if you have multiple publication offers.
Cite Yourself?
Depending on how your citations are being judged, citing yourself may
seem like an easy and attractive way to boost your stats. Hein currently
includes self-citations in its overall author citation counts.
149
However,
146
Harrison & Mashburn, supra note 55, at 76-77.
147
See Submitting Papers to Law Journals: Journal Rankings, U. MICH. L. LIBR.,
https://libguides.law.umich.edu/journal_submissions/rankings [https://perma.cc/232Z-VC49]
(providing a list of journal ranking metrics, which the four journals listed consistently top).
148
Harrison & Mashburn, supra note 55 at 76-77.
149
Shannon Furtak, Author Self-Citations, ScholarCheck Rankings, and Twitter Buzz,
HEINONLINE: BLOG (Dec. 19, 2017), https://home.heinonline.org/blog/2017/12/author-self-
citations-scholarcheck-rankings-and-twitter-buzz/ [https://perma.cc/QBC6-TFWU] (“Self-
citations are also not subtracted from overall citation counts and are there for informational
purposes only.”).
214
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Hein does separate out self-citations.
150
This means that your institution
and U.S. News or other citation rankers can easily remove them if
desired. Further, should someone take it upon themselves to investigate
self-citation, as has happened in the scientific field, spurious self-citers
will be found out.
151
In the SEO world, Google sometimes punishes
websites that have “unnatural links” by suppressing them in results.
152
Unnatural links would be similar to an author unnaturally citing
themselves or having a quid pro quo citation arrangement with a
colleague. Although these manipulative tactics may be successful in the
short term, we suspect they will eventually be discovered either by your
colleagues (as happened in the scientific community) or by U.S. News
or an institution making tenure and other citation-influenced
decisions.
153
If this happens, the fallout will likely be serious. Therefore,
we only recommend self-citation if your article is truly the best source
to support your claim.
150
HEINONLINE, https://heinonline.org/HOL/Welcome [https://perma.cc/W45G-RX7S] (visit
the Hein author profile of a well-cited author, like Cass Sunstein’s, and you will see the
citation statistics include a “self-citations” line item).
151
See, e.g., John P. A. Ioannidis, Jeroen Baas, Richard Klavans & Kevin W. Boyack, A
Standardized Citation Metrics Author Database Annotated for Scientific Field, PLOS BIOL. 1,
1 (Aug. 12, 2019), https://doi.org/10.1371/journal.pbio.3000384 [https://perma.cc/KT5L-
5S2S] (“Citation metrics are widely used and misused. We have created a publicly available
database of 100,000 top scientists that provides standardized information on citations, h-index,
coauthorship-adjusted hm-index, citations to papers in different authorship positions, and a
composite indicator.”).
152
Link Schemes, GOOGLE DEVELOPERS, https://developers.google.com/search/docs?hl=en
[https://perma.cc/ZE69-VZ8Y] (follow “Advanced SEO” hyperlink; then open “Guidelines”
dropdown menu; then open “Quality guidelines” dropdown menu; then follow “Link
schemes” hyperlink).
153
See, e.g., Eric A. Fong & Allen W. Wilhite, Authorship and Citation Manipulation in
Academic Research, PLOS ONE 1 (Dec. 6, 2017),
https://doi.org/10.1371/journal.pone.0187394 [https://perma.cc/FT5U-KMCX] (discussing
scholars in other fields who include non-contributing authors, add citations not relevant to
their work, etcetera).
2021]
WILLEY & KNAPP
215
Publish with A Well-Cited Author
As you would expect, well-cited authors were more likely to appear in
our top-articles sample where the average citation count for the first
author was 1368, compared to 622 for the bottom-articles, author over
100 citations sample, and 437 of our bottom-articles, 3 citations
sample. Therefore, you stand to increase your chances of citation by
working with an already well-established author, assuming each author
listed on an article receives credit for one citation per article (below we
discuss why this system may invite issues). However, we want to point
out that this is only one factor that can help you, not a guarantee: Cass
Sunstein (one of the most cited authors on Hein) had five articles in our
top-articles sample, but also had two in the bottom-articles, 3
citations sample.
Publish with a Co-Author?
At 1.4, our top-articles sample had the highest average number of
authors per article, compared to 1.36 in bottom-articles, author over
100 citations
154
and 1.27 in bottom-articles, 3 citations.
155
However,
our analysis indicates that only the difference between 1.4 and 1.27 is
statistically significant. We were surprised when we discovered the
difference was only significant between one of our lower citation
datasets. We expected that more authors would mean more people
promoting the article, increased chances for name recognition, and,
thus, more citations. In their study, Ayers and Vars found that
coauthored pieces were cited more often, but articles with more than
two authors were cited less frequently.
156
Further, multiple studies
looking at other disciplines found that more authors correlated to more
154
Not statistically significant.
155
Statistically significant.
156
Ayres & Vars, supra note 21, at 439.
216
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
citations.
157
In non-legal fields, we found only one study that indicated
more authors led to fewer citations.
158
Even this study found that
overall, more authors led to more citations, but in a few of the studied
disciplines articles with fewer authors did better.
159
So why were our
findings mixed in this area? It’s possible that the legal field is unique
and there is no correlation between number of authors and citations.
However, we think it is more likely that the correlation exists but is
weaker than the others we studied, and, if we analyzed more data, we
would find that multiple authors do increase the chances for citation.
Check Hein Records and Ensure Credit for Your Work with
ORCID
On a purely practical note, we have observed Hein sometimes
mistakenly includes or excludes works by authors with similar names
from its citation metrics. To make sure that your work is not credited to
someone with a similar name, we recommend that you check your CV
against what is shown on your Hein Author Profile. You should also use
a persistent digital identifier, like ORCID.
160
Whenever possible, tie
your scholarship to your ORCID iD. This is especially important if you
have a common name. Additionally, Hein has integrated ORCID with
its author profiles, allowing authors to push works on their Hein profile
to their ORCID account.
161
More importantly, Hein recently gave
157
E.g., van Wesel et al., supra note 1, at 1608-12, and David Card & Stefano DellaVigna,
Nine Facts About Top Journals in Economics, 51 J. ECON. LITERATURE 144, 156 (2013). But
see John Hudson, Be Known by the Company You Keep: Citations—Quality or Chance?, 71
SCIENTOMETRICS 231, 234 (2007) (finding no impact on number of authors in a study of two
economic journals).
158
Massimo Franceschet & Antonio Costantini, The Effect of Scholar Collaboration on
Impact and Quality of Academic Papers, 4 J. INFORMETRICS 540, 544 (2010) (looking at
scholarly work at Italian Universities across twenty disciplines).
159
Id. (noting, for example, that in physics, where works with over 100 authors are common,
fewer authors led to more citations).
160
ORCID, https://orcid.org/ [https://perma.cc/X9MD-7KUZ].
161
Creating an ORCID Account and Linking to HeinOnline, HEINONLINE: KNOWLEDGE BASE,
https://help.heinonline.org/kb/creating-an-orcid-account-and-linking-to-heinonline/
[https://perma.cc/LT8U-BA9G].
2021]
WILLEY & KNAPP
217
authors the ability to pull works from their ORCID accounts to their
Hein author profiles.
162
This allows authors to bring in scholarship that
is unavailable in Hein’s database, like books and interdisciplinary
articles.
163
Eventually, Hein hopes to add citations to these works that it
finds in its database to an author’s total citation counts.
164
If Hein starts
tracking these additional citations, it could lead to a significant increase
in some authors’ overall citation counts. Bonnie Shucha authored an
article that goes into greater detail on the items covered in this
paragraph.
165
We recommend reviewing it for more information.
More SEO Practices to Consider
Although we do not have related data from our study, we wanted to
share a few additional practices and tips from the SEO world that may
help academic authors. This is certainly not an exhaustive list. We
recommend that you mine the SEO world for additional ideas on how
to increase your citation counts.
Other academic fields are already looking at how to optimize
scholarship for search engines,
166
even coining the phrase “academic
162
Id.
163
Why Hein’s ORCID Integration Could Be a Big Deal for the US News Law School
Scholarly Impact Ranking, WISBLAWG (Feb. 5, 2020),
https://wisblawg.law.wisc.edu/2020/02/05/why-heins-orcid-integration-could-be-a-big-deal-
for-the-us-news-law-school-scholarly-impact-ranking/ [https://perma.cc/RBB6-ZH92].
164
Id.
165
Shucha, supra note 23.
166
E.g., Sharon Shafer, SEO for Authors: A How-to-Guide, UCLA LIBR.: RES. VISIBILITY,
https://guides.library.ucla.edu/seo/author [ https://perma.cc/Y7YH-WF5C]; Search Engine
Optimization: For Authors, WILEY, https://onlinelibrary.wiley.com/pb-
assets/assets/17476593/SEO_Guidelines_for_Authors_July_13-1509465663000.pdf; It's All
About Getting Found! Search Engine Optimization Tips for Book Authors, SPRINGER NATURE,
https://www.springernature.com/gp/authors/campaigns/seo-tips-for-book-authors
[https://perma.cc/NV83-Y3ZG].
218
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
search engine optimization.”
167
(We think we are the first to come up
with “academic citation optimization”
168
and “legal citation
optimization.”) Much of the work in other fields has focused on
optimizing within Google Scholar.
169
Because Hein and other databases
allow Google Scholar to index their content (researchers can access
results for their institution by editing their settings in Google Scholar),
this seems an area legal practitioners should pay attention to. In the
future, we believe SEO tactics will become increasingly important for
scholars in all fields and will also grow more nuanced with each
discipline doing analysis and providing insights on how to rank in the
discipline’s most popular databases. We hope to begin this in the legal
field by applying SEO principles to some popular legal databases.
170
When writing, include all variants of words users may search for and
put keywords in the title. While Google is getting better and better at
recognizing and including results for synonyms, many legal search
engines still struggle in this area. As one example, we ran a test on June
10, 2020, in Westlaw Edge. We wanted to find an article we have
frequently cited here: Determinants of Citations to Articles in Elite Law
Reviews.
171
First, we entered “determinants of citations journals”
(without quotes or bold). The desired article came up at number 15 in
the results, under the “secondary sources” content type, when sorted by
“relevance” (the default sorting method). Simply by changing the search
to “determinants of citations reviews” (without quotes or bold), the
article moved to the top position with the same filters. Perhaps if the
167
Lluís Codina, Received Citations as a Main SEO Factor of Google Scholar Results
Ranking, SEO MEDIA LAB (June 21, 2018), https://seomedia-lab.com/academic-seo-google-
scholar/ [https://perma.cc/7VNY-27UN].
168
We have started a site for general academic citation optimization,
(https://www.citationoptimization.com/) and hope to start one focused on legal citation
optimization (legalcitationoptimization.com).
169
Codina, supra note 167 (finding that number of citations is the “most relevant off-page
feature in the ranking of search results on Google Scholar”).
170
If time permits, we hope to build on what we cover here and elsewhere in this paper on a
website at legalcitationoptimization.com.
171
Ayres & Vars, supra note 21.
2021]
WILLEY & KNAPP
219
article were titled “Determinants of Citations to Articles in Elite Law
Reviews & Journals,” it would have shown first for both searches.
172
Related to this, we also recommend thinking carefully about keywords
and variations when you upload to repositories like SSRN. If someone
else handles your SSRN uploads, make sure to include a set of relevant
keywords when you send them your manuscript. Whoever does the
actual uploading will undoubtedly do their best, but no one knows your
article as well as you do.
You may also want to consider citing well-known articles in your
article. An SEO study found that outgoing links
173
to authoritative sites
have a positive impact on rank in Google’s search results.
174
Applying
this to scholarly writing would mean citing to articles that already have
high citation numbers. There are two reasons this approach has
potential. First, advanced algorithms in legal databases may work
similar to Google and give positive weight simply because your article
cites to an already well-cited article.
175
Second, Westlaw has a “citing
references” tab and Lexis offers “other citing sources.”
176
Both of these
list articles that cite the one being viewed. So, by citing to a popular
article, you will automatically place your article on these pages. These
tools are heavily used and something that we encourage our students to
review whenever they find a particularly relevant source. Westlaw
defaults to ranking these by depth of treatment, so you could go a step
further and cite heavily to a popular article so that yours appeared at the
172
If you are curious, Google pulled up the desired article first with “reviews” and second
with “journals.” Search run at www.google.com on June 12, 2020.
173
Outgoing links are links within the content of one domain to content on another domain.
174
E.g., Shai Aharony, Study – Outgoing Links Used as Ranking Signal, ELITE DIGITAL
MARKETING, https://www.rebootonline.com/blog/long-term-outgoing-link-experiment/
[https://perma.cc/X9FN-Z7KT].
175
Id.
176
When viewing a case on LexisNexis, the “other citing sources” tab appears in the center of
the top of the page. When viewing a case on Westlaw, the “citing references” tab appears in
the center of the ribbon of the page.
220
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
top of the results for that article. Of course, this only works if your
article is actually on a topic that people looking at the article you cited
will find interesting. Also, we need to point out that the 2000 Ayres and
Vars study found that fewer footnotes-per-page correlated to an increase
in citations, so too many citations may hurt you.
177
(Studies outside the
legal field have found the opposite.)
178
Another approach advocated in SEO circles is to take a contrarian
position on a popular topic.
179
In the legal field, Brian Leiter has
identified this type of work as likely to be cited, “some work is so
wrong, or so bad, that everyone acknowledges it for that reason.”
180
Here, you would write on a popular topic that already has a largely
accepted viewpoint, but your article would take the opposite of the
accepted position. Trying to compete with established articles while
espousing the accepted viewpoint will prove extremely challenging.
Authors who take a less obvious or unpopular position instantly reduce
their field of competition, while still allowing themselves to cover a
popular topic. Additionally, these articles can gain citations from
177
Ayres & Vars, supra note 21, at 439.
178
Zoë Corbyn, An Easy Way to Boost a Paper’s Citations, NATURE (Aug. 13, 2010),
https://doi.org/10.1038/news.2010.406 [https://perma.cc/47FF-FRH9] (discussing a 2010
study with its author: Gregory D. Webster, Scientists Who Cite More Are Cited More:
Evidence from over 50,000 Science Articles). But see Phil Davis, Reference List Length and
Citations: A Spurious Relationship, SCHOLARLY KITCHEN (Aug. 18, 2010),
https://scholarlykitchen.sspnet.org/2010/08/18/reference-list-length-and-citations-a-spurious-
relationship/ [https://perma.cc/3FWA-RCAQ] (stating that Webster’s purported correlation
was due to other factors, a possibility at which Webster hinted in his presentation).
179
Elisa Gabbert, Contrarian Content Marketing: How to Zig When Everyone Else Zags,
WORDSTREAM (Aug. 9, 2017), https://www.wordstream.com/blog/ws/2014/05/19/contrarian-
content-marketing [https://perma.cc/GN8M-CXBN]. See also Smith, supra note 31, at 340-41
(noting that it may be that case law decisions that depart from established doctrine have a
better chance of becoming influential—i.e., being cited too often).
180
Brian Leiter, Brian Leiter's Top 40 Faculties Based on Per Capita Scholarly Impact
(Citations), 2003-04, BRIAN LEITER’S L. SCH. RANKINGS (July 16, 2003),
http://www.leiterrankings.com/faculty/2003faculty_impact_cites.shtml
[https://perma.cc/J9M5-93XH]; see also Ayres & Vars, supra note 21 (“Articles with many
citations might be high quality or low quality (as other authors go out of their way to criticize
the argument)”).
2021]
WILLEY & KNAPP
221
authors simply looking to present both sides of a position, and, in some
cases, may even prompt articles focused solely on refuting the
contrarian position. In the latter case, the old adage “no press is bad
press” applies, and any articles refuting yours will inevitably need to
cite your work (see below for a proposal to allow authors to prevent
certain citations from counting towards other authors).
Lessons from SEO for U.S. News’ Scholarly Impact Ranking
181
In 2019, U.S. News announced that it may publish a new ranking that
evaluates the scholarly impact of U.S. law schools.
182
U.S. News
indicated that initially they will make this ranking separate from the
Best Law Schools ranking.
183
Originally, U.S. News slated the scholarly
impact ranking for 2019,
184
they then updated that sometime in 2020,
185
and, in late 2020, revised again to sometime in 2021.
186
The reason for
the delay is not clear, but Hein is actively working to improve its system
181
U.S. News has yet to make an official announcement, but they have privately indicated that
they will likely not move forward with a scholarly impact ranking at this point. However, we
opted to keep this section because it remains relevant should U.S. News or others later decide
to move forward with a scholarly impact ranking. Brian Leiter, US News Is *Not* Going to
Produce Its Own Scholarly Impact Rankings Using HeinOnLine, BRIAN LEITER'S L. SCH. REP.
(August 20, 2021), https://leiterlawschool.typepad.com/leiter/2021/08/us-news-is-not-going-
to-produce-its-own-scholarly-impact-rankings-using-heinonline.html [https://perma.cc/L97X-
CKR3].
182
Robert Morse, U.S. News Considers Evaluating Law School Scholarly Impact, U.S. NEWS
(Feb. 13, 2019, 1:00 PM), https://www.usnews.com/education/blogs/college-rankings-
blog/articles/2019-02-13/us-news-considers-evaluating-law-school-scholarly-impact
[https://web.archive.org/web/20200410224129/https://www.usnews.com/education/blogs/coll
ege-rankings-blog/articles/2019-02-13/us-news-considers-evaluating-law-school-scholarly-
impact].
183
Id.
184
Id.
185
Karen Sloan, Changes to US News Law School Rankings Met with Skepticism, LAW.COM
(Jan. 8, 2020, 2:42 PM), https://www.law.com/2020/01/08/changes-to-u-s-news-law-school-
rankings-met-with-skepticism/ [https://plus.lexis.com/api/permalink/bf32843a-b785-46bf-
bd27-67c31c4a4252/?context=1530671].
186
Caron, U.S. News to Publish, supra note 22 (publishing an email stating that U.S. News
plans to publish its first scholarship ranking in 2021 and will use Hein).
222
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
— perhaps that has something to do with it.
187
Here, we discuss the new
ranking, its implications, and opportunities for the types of abuse that
plagued the early SEO world. Everything we discuss remains applicable
regardless of whether U.S. News eventually incorporates scholarly
impact into its overall law school ranking, as some suspect they will,
188
or keeps it separate. Even if U.S. News never produces this sort of
ranking at all, the items covered here would still apply to others
involved in citation rankings. Rankings matter to law schools.
189
However, the pressures and incentives we mention will increase if
scholarly impact becomes part of the overall U.S. News law school
ranking. Especially if it replaces the weighty “peer-assessment”
score,
190
as would seem logical.
U.S. News does need a better system.
191
The most heavily weighted
metric in the current system is the subjective “peer assessment score,”
accounting for 25 percent of the overall rank.
192
As you would suspect,
187
See Why Hein’s ORCID Integration Could Be a Big Deal for the US News Law School
Scholarly Impact Ranking, WISBLAWG (Feb. 5, 2020),
https://wisblawg.law.wisc.edu/2020/02/05/why-heins-orcid-integration-could-be-a-big-deal-
for-the-us-news-law-school-scholarly-impact-ranking/ [https://perma.cc/RBB6-ZH92].
188
Sloan, supra note 185 (“Since then, he has received an earful from law professors with
concerns not only about how the ranking will be calculated, but about its potential to influence
law faculty hiring, particularly if scholarly impact scores are eventually incorporated into the
overall law school ranking.”).
189
Joe Patrice, This Is Why the Rankings Matter, ABOVE L. (Mar. 13, 2019, 10:43 AM),
https://abovethelaw.com/2019/03/this-is-why-the-rankings-matter/ [https://perma.cc/8BKR-
CGVU].
190
Robert Morse, Ari Castonguay & Juan Vega-Rodriguez, Methodology: 2021 Best Law
Schools Rankings, U.S. NEWS (Mar. 16, 2020), https:/www.usnews.com/education/best-
graduate-schools/articles/law-schools-methodology
[https://web.archive.org/web/20210311181154/https:/www.usnews.com/education/best-
graduate-schools/articles/law-schools-methodology] (the peer-assessment score currently
accounts for 25 percent of the ranking).
191
See Bernard S. Black & Paul L. Caron, Ranking Law Schools: Using SSRN to Measure
Scholarly Performance, 81 IND. L.J. 83, 87, 111 (2006) (discussing issues with the current
U.S. News Peer Reputation metric and comparing it to other systems).
192
See Paul J. Heald & Ted Sichelman, Ranking the Academic Impact of 100 American Law
Schools, 60 JURIMETRICS J. 1, 2 (2019), and Morse, Castonguay & Vega-Rodriguez, supra
note 190.
2021]
WILLEY & KNAPP
223
U.S. News’ influence shows up in the peer assessment score, at least
within the upper echelons, which roughly track the U.S. News
rankings.
193
However, when you move outside the top twenty
significant differences begin to appear. For example, Oregon is ranked
83
rd
overall in the 2020 rankings, while its peer assessment would have
placed it at 52
nd
.
194
Temple ranks 48th overall and its peer assessment
puts it at 57
th
.
195
This makes one wonder what peers are basing their
rankings on. If the other, mostly objective measures, U.S. News uses to
rank schools placed Oregon 35 spots below Temple, on what basis does
the peer assessment rank Oregon above Temple? How are they even
close in the peer ranking? It seems likely that people pay close attention
to the top part of the U.S. News scores, which then feeds into their peer
assessment score and helps keep the top portion of the rankings
relatively stagnant. Meanwhile, assessors likely pay less attention to
lower ranked schools like Oregon and Temple and, therefore, make their
assessment somewhat randomly. Replacing part or all of the peer
assessment with a scholarly impact score would remove some of the
subjectivity that currently plagues it. Instead of asking scholars and
administrators to randomly rank their favorite schools, the scholarly
impact score would use scholars’ citations to evaluate how they feel
about their peers. Generally, scholars cite those they think have done
good work (see below for a suggestion to exclude citations when the
author does not agree with what they cite). Further, the articles authors
cite are not anonymous, and citing low-quality sources can negatively
impact an author’s reputation. This creates an incentive for authors to
invest time vetting the sources they cite and holds them accountable for
failing to do so. These incentives are absent in the current U.S. News
system. In fact, the current incentives encourage participants to vote in
193
Black & Caron, supra note 191, see also Paul Caron, 2020 U.S. News Law School Peer
Reputation Rankings (And Overall Rankings), TAXPROF BLOG (Mar. 12, 2019)
https://taxprof.typepad.com/taxprof_blog/2019/03/2020-us-news-law-school-peer-reputation-
rankings-and-overall-rankings.html [https://perma.cc/5V2N-WCJW] [hereinafter Caron, 2020
U.S. News].
194
Caron, 2020 U.S. News, supra note 193.
195
Id.
224
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
the way that most benefits them. For these reasons, we believe that
moving to a citation-based ranking would be an improvement over the
current peer assessment score.
However, using Hein to rank scholarship also raises several questions.
To start, some believe the scholarly impact rankings may not change the
rank of most schools.
196
Even if this is the case for most schools, which
it may or may not be depending on whether the prognosticators have
accurately predicted what U.S. News might actually do, for those
schools who are impacted it would be worth the change. Thus, we think
that this is not a valid reason for discarding a citation-based approach
and other potential issues deserve more focus.
Factors Beyond Quality May Impact Citations
As our study demonstrates, factors beyond the quality of an article may
impact citations. If the goal is for U.S. News’ new ranking is to more
closely track the quality of schools, the results of our study may weigh
against the proposed solution. However, our study did not evaluate the
actual quality of the articles, so it is possible that higher-quality articles
are more likely to have the characteristics we looked at. In other words,
maybe we found a way to identify high-quality articles. Given the
number of articles we looked at, it seems more likely that these factors
do impact citations independent of quality. If that is the case, then
articles like ours will eventually level the playing field by making clear
to motivated authors how they can give their article the best chance to
get cited. Further, because our study found several well-cited articles
that did not fall within optimal ranges, it seems clear that truly
exceptional articles will still gain more citations. In sum, we do not
believe that our findings should dissuade U.S. News from implementing
their new ranking system.
196
Derek T. Muller, Gaming Out Hein Citation Metrics in a USNWR Rankings System,
EXCESS DEMOCRACY (Feb. 25, 2019), https://excessofdemocracy.com/blog/2019/2/gaming-
out-hein-citation-metrics-in-a-usnwr-rankings-system [https://perma.cc/23SB-RYMW].
2021]
WILLEY & KNAPP
225
Increased Pressure Leads to Stronger Motives for Abuse
Most of the potential issues we discuss next stem from the added
pressure that the new ranking system would apply to law schools. Other
disciplines already deal with this type of pressure.
197
In at least one case,
attempts to manipulate rankings have led to criminal charges.
198
This
increased pressure is also evidenced by the significant number of
articles on factors influencing citations in non-legal fields
199
and by
some of the negative practices, like “coercive citation,” that occur with
surprising regularity in other fields.
200
Further, how to “game” ranking
systems has become a popular topic in interdisciplinary ranking systems
like Google Scholar.
201
More troubling, scientists in a focus group said,
“competition contributes to strategic game-playing in science, a decline
in free and open sharing of information and methods, sabotage of
others’ ability to use one’s work, interference with peer-review
197
See Georg Franck, Scientific Communication – A Vanity Fair?, 286 SCI. 53 (1999)
(discussing how scientists’ desire for citations motivates the formation of “citation cartels”),
and Maarten van Wesel, Evaluation by Citation: Trends in Publication Behavior, Evaluation
Criteria, and the Strive for High Impact Publications, 22 SCI. & ENG’G ETHICS 199, 203
(2016) [hereinafter van Wesel, Evaluation by Citation] (listing tactics employed in other fields
to increase citation counts, including: “artificially inflating the author count, adding
unnecessary references, and purposely making the abstract hard to read”).
198
Scott Jaschik, Ex-Dean at Temple Indicted on Charges of Manipulating Rankings, INSIDE
HIGHER ED (April 19, 2021),
https://www.insidehighered.com/admissions/article/2021/04/19/ex-dean-temple-indicted-
charges-manipulating-rankings [https://perma.cc/9VFP-ME5P].
199
See, e.g., van Wesel et al., supra note 1 (discussing causes of citations and collecting
citations to prior studies).
200
Dalmeet Singh Chawla, Two-thirds of Researchers Report ‘Pressure to Cite’ in Nature
Poll, NATURE (Oct. 1, 2019), https://doi.org/10.1038/d41586-019-02922-9
[https://perma.cc/H78S-YEA4].
201
Emilio Delgado López-Cózar, Nicolás Robinson-García & Daniel Torres-Salinas,
Manipulating Google Scholar Citations and Google Scholar Metrics: Simple, Easy and
Tempting (Cornell Univ. Computer Sci., EC3 Working Papers 6, 2012),
https://arxiv.org/abs/1212.0638 (demonstrating how researchers can easily manipulate their
Google Scholar h-index); How to Successfully Boost Your H-Index, ENAGO ACAD. (Nov. 4,
2019), https://www.enago.com/academy/how-to-successfully-boost-your-h-index/
[https://perma.cc/6XL6-3BFN] (discussing more legitimate ways authors can improve their h-
index).
226
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
processes, deformation of relationships, and careless or questionable
research conduct.”
202
We now discuss similar issues and their parallels
from the SEO world that could arise in the legal field if U.S. News
updates its system.
Citation Cartels
Hein’s current system gives one citation to each author regardless of the
number of authors.
203
For example, if an article has twenty citations and
lists author A and author B, then both authors receive twenty citations
towards their total citation count, meaning that article is counted for
forty total citations. Whether U.S. News’ co-author crediting system will
mirror Hein’s is not clear. If it does, the system will incentivize scholars
to increase the number of co-authors. Following Hein’s method for
distributing citations will also incentivize bad behavior.
To prevent helping competitors in the rankings, law schools may
encourage their professors to only co-author with scholars from the
same institution. This would allow schools to get two citations for a
single article where they would only get one citation if the co-author
worked at a different institution. Taking this a step further, especially
close competitor schools may be specifically embargoed, while peers at
schools distant within the rankings allowed. Inhibiting potentially
beneficial collaborations for the sake of rankings would negatively
impact legal scholarship.
In the SEO world, websites sometimes make agreements to link to each
other with the goal of improving the rank of all sites involved.
204
And,
although it is rare now, website owners used to pay other websites to
202
Melissa S. Anderson et al., The Perverse Effects of Competition on Scientists’ Work and
Relationships, 13 SCI. & ENG’G ETHICS 437, 437 (2007).
203
Sloan, supra note 185.
204
Link Schemes, supra note 152.
2021]
WILLEY & KNAPP
227
link to their site.
205
Similarly, legal authors may start making
agreements to include each other as co-authors, regardless of whether
the other made any meaningful contribution. Researchers have already
identified this type of behavior in other fields. Back in 1981, William
Broad discussed studies and anecdotal evidence indicating that
pressures to produce citations were leading to the gratuitous addition of
names to articles.
206
Further, studies have identified “citation cartels” in
other fields.
207
Although it has not been studied, this behavior may exist
in legal circles as well. By adding its metric, U.S. News will only
increase pressure on authors to create these types of agreements with
each other. Taken to the extreme, perhaps schools or authors may even
pay for citations as was once common in the SEO world. Clearly, these
types of behaviors will damage the quality of scholarship by giving
benefits to authors for reasons beyond their merit.
The one positive development that we will see if U.S. News follows
Hein here is an increase in legitimate collaboration. Still, we think that
the potentials for abuse should lead U.S. News to create a system where
citations are distributed between authors. Further, in true co-author
relationships, the work is shared. This means that those who co-author
should need to spend less time per article, allowing them to publish
more total articles than a scholar who does not write with co-authors.
So, fairness may also dictate that U.S. News somehow split the citations
that co-authored articles receive. It could be an even split where all
authors get the same percentage of each citation, or it could be weighted
in some way — perhaps by where the author’s name appears.
208
205
Id.
206
William J. Broad, The Publishing Game: Getting More for Less, 211 SCI. 1137, 1137
(1981).
207
Iztok Fister Jr., Iztok Fister & Matjaž Perc, Toward the Discovery of Citation Cartels in
Citation Networks, 4 FRONTIERS PHYSICS 1 (2016).
208
Other fields are already discussing various solutions to this issue, e.g., John P. A.
Ioannidis, Measuring Co-Authorship and Networking-Adjusted Scientific Impact, PLOS ONE
1 (July 28, 2003), https://doi.org/10.1371/journal.pone.0002778 [https://perma.cc/8RQM-
YYWU].
228
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Opportunities for Top Journals to Favor Their Scholars
As shown in our data, top journals tend to get more citations. In a world
focused on scholarly rankings, might law schools pressure their journal
editors to publish their own scholars? Perhaps this already happens,
209
but could a new ranking system make it worse? While schools that
already hold a top rank and a top journal, like Harvard and Yale, may
have less incentive to do this — they cannot rank any better and the
fallout from undue influence could be significant — other schools with
a lower rank and well cited journals may feel pressure. For example, the
Washington and Lee Law Journal Rankings, found that the Iowa and
Fordham law reviews received the 8
th
and 9
th
most citations in 2019.
210
U.S. News ranked the University of Iowa 27
th
and Fordham 37
th
.
211
For
schools in this situation, the pressure to favor their scholars would be
strong and, potentially, fully within the bounds of U.S. News’s
requirements. Yaniv Reingewertz and Carmela Lutmar have
documented that this already occurs in other disciplines.
212
With the
change to U.S. News rankings, law schools may follow suit.
To address this, U.S. News could do an analysis and determine how
often journals tend to publish their own authors now. Based on this
average, U.S. News could set a threshold for articles published in home
journals. U.S. News would then not count, or give decreased weight, to
articles published in the home journal above the threshold.
209
See Ayres & Vars, supra note 21, at 442 (finding statistically significant evidence that after
1990 journals tend to favor their own scholars).
210
W&L Law Journal Rankings, supra note 140.
211
Staci Zaretsky, Behold, The FULL 2019 U.S. News Law School Rankings Leak (1-144),
ABOVE L. (Mar. 14, 2018, 9:28 PM), https://abovethelaw.com/2018/03/behold-the-full-2019-
u-s-news-law-school-rankings-leak-1-144-rnp/ [https://perma.cc/4F8S-2ENH].
212
Yaniv Reingewertz & Carmela Lutmar, Do Academic Journals Favor Researchers from
Their Own Institutions?, HARV. BUS. REV. (Feb. 27, 2018), https://hbr.org/2018/02/do-
academic-journals-favor-researchers-from-their-own-institutions [https://perma.cc/SN78-
G3Y3].
2021]
WILLEY & KNAPP
229
Keyword Stuffing
Keyword stuffing was a popular black hat SEO method where website
owners would load their content with keywords they wanted to rank
for.
213
This could mean simply including the keyword in the text more
often than natural
214
or it could mean hiding keywords in the code so
that Google’s bots see them but human visitors do not.
215
At one point,
this method worked well and sites that adopted it were able to push
themselves up in Google’s results.
216
Of course, Google eventually
caught on to this practice and essentially penalized websites that used
it.
217
As touched on earlier, most legal databases lack the sophisticated search
capabilities Google has. This means they are likely susceptible to tactics
similar to keyword stuffing. The editorial process of most journals will
make this harder to pull off in law reviews, but it remains possible.
Rather than damage the quality of their main content, perhaps authors
will stuff footnotes with keywords they want to rank for.
There is not much U.S. News can reasonably do to counter this
possibility. However, databases will eventually adjust their algorithms,
as Google did, making their systems better overall. So, while early
adopters may see a temporary bump in their ranking, we do not think
this will be a long-term problem.
SEO Experts & Law Schools
In an effort to game the system, law schools may hire SEO-type experts
to analyze and figure out legal database algorithms, then instruct faculty
213
Irrelevant Keywords, supra note 81.
214
Id.
215
Hidden Text and Links, GOOGLE DEVELOPERS: GOOGLE SEARCH CENT.,
https://support.google.com/webmasters/answer/66353?hl=en&ref_topic=6001971
[https://perma.cc/7SJR-T2FY].
216
Marrs, supra note 61.
217
Irrelevant Keywords, supra note 81.
230
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
on how to write to those algorithms. The work of SEO experts in
providing insights into Google’s extremely complex and guarded
algorithm indicates that deconstructing the less-complicated legal
database algorithms is certainly feasible. If this happens, it could track
the development of the SEO world. Initially, the experts would find
easily exploitable elements of the algorithms and suggest questionable
methods to increase rank within the databases and, consequently,
chances for citations. As legal databases plug the holes, the focus would
return to more reasonable methods, similar to those SEO experts
recommend today. The question would be whether the fixes legal
databases make to their algorithms have the effect of punishing the
articles, schools, and scholars that took advantage of the holes. Since
U.S. News’ current proposal would look back only five years, the impact
of any “punishment” may be limited to that time frame. Less fear of
retribution could make schools more likely to engage in questionable
practices. U.S. News could reduce incentives for this; pulling articles
from a longer time period — say ten years.
Ultimately this may not be a negative development anyway. The impact
on the scholarship itself would depend on the recommendations the
citation experts offer. If they suggest things like using more headings to
make skimming easier, scholarship should not be negatively impacted.
On the other hand, suggestions to artificially work in more keywords
would detract from the scholarship, as touched on above.
Decrease in Important but Less Popular Topics
Earlier, we shared data indicating that popular topics tend to gain more
citations. And we recommended incorporating popular topics into areas
of the law authors are interested in to increase citations. However, if our
advice is taken a step too far, scholars will shift the entire area of the
law they focus on, moving from tax to constitutional law in hopes of
2021]
WILLEY & KNAPP
231
increasing citations.
218
Setting aside the negative impact this may have
on the scholars themselves, it will lead to a decrease in scholarship on
the important but less high-profile topics the legal world is full of.
Jeff Sovern takes this idea a step further, hypothesizing that schools will
start choosing faculty based on how many scholars write on a subject,
“[t]o be more concrete, imagine that a law school is hiring a new
professor and has two candidates. One candidate writes about criminal
law and the other writes about consumer law. The law school wants to
maximize its ranking, and so wants to hire the candidate whose work
will be cited more. The universe of people writing scholarly articles
about criminal law is much larger than the universe of professors
writing about consumer law, and so, all other things being equal, the
criminal law professor is likely to rack up more citations and so help
with the school's ranking more.”
219
Clearly, this would be a negative
development for the legal education system and the legal world as a
whole.
U.S. News could avoid this issue by incorporating a field-weighted
metric to normalize citations across areas of the law. Given the potential
loss, we certainly hope that U.S. News (and anyone else involved in
citation-based rankings) implements this sort of system. Other fields
already have systems like these in place,
220
so, this certainly seems
technically feasible.
218
Leiter, supra note 180 (“Law reviews publish lots on constitutional law, and very little on
tax”).
219
Jeff Sovern, How the New US News Scholarly Impact Ranking Could Hurt Niche Subjects,
Like Consumer Law, CONSUMER L. & POL’Y BLOG (Mar. 7, 2019),
https://pubcit.typepad.com/clpblog/2019/03/how-the-new-us-news-scholarly-impact-ranking-
could-hurt-niche-subjects-like-consumer-law.html [https://perma.cc/FR5D-GFJ2].
220
E.g., What is Field-Weighted Citation Impact (FWCI)?, SCOPUS SUPPORT CTR. (Sept. 16,
2020), https://service.elsevier.com/app/home/supporthub/scopus [https://perma.cc/8GUS-
HHXY] (follow “Content” hyperlink; then follow “What is Field-weighted Citation Impact
(FWCI)?” hyperlink).
232
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
Do Citations Follow the Scholar?
A question U.S. News has yet to answer is what happens when a scholar
moves schools? Presumably, any future citations would be attributed to
the new school but what about citations from works written at the old
school? Currently, Hein does not designate where authors were
employed when they received their citations. Could law school deans
lend well cited authors to their friends as visiting scholars? Or, might
schools in need of a quick bump pay exorbitant short-term salaries to
bring over scholars for a few years so they can benefit from their citation
counts at critical moments? U.S. News should address this potential
problem by clearly stating that citations follow the school, not the
scholar, when it comes to their ranking system. The feasibility of this
will depend on whether Hein can develop a method to track where
faculty are employed when they author articles.
Publishing the Same Content in Multiple Places
In some fields, authors publish the same or similar articles in multiple
places, thus artificially increasing their publication counts.
221
This
would clearly allow faculty to increase the number of articles they
publish. In theory, citations could also potentially increase by exposing
the content to a wider audience. However, this approach may backfire
in a system based on number of citations because the duplicate pieces
will likely end up competing. This will prevent either article from
gaining broad acceptance or ever becoming the article on a certain topic.
It could also confuse databases and lead to lower ranks in search results,
221
See Mounir Errami & Harold Garner, A Tale of Two Citations, 451 NATURE 397, 397-98
(2008) (stating that 4.7 percent of bio-medical researchers surveyed admitted to repeated
publication of the same results and finding that duplicate articles increased significantly
between 1975 and 2005).
2021]
WILLEY & KNAPP
233
as happens with Google’s algorithm.
222
If both duplicate articles end up
buried in database results, many scholars will never even see them.
Further, this type of behavior could easily be identified either through
modifications to database algorithms or with existing tools designed to
find instances of plagiarism. Since the benefits of this type of behavior
remain unclear, it is less likely that scholars will adopt it. Additionally,
for scholars currently judged by total output rather than citations, the
new ranking system may encourage more original work. Here our
recommendation is for authors rather than U.S. News: avoid duplicating
your work to prevent the issues we discuss.
Lack of Coverage of Non-Legal Sources
Hein’s focus is on legal materials. As others have pointed out, this
means it is likely to miss citations to legal articles in non-legal sources,
and, even worse, the existence of non-legal articles written by law
professors.
223
It also means that citations to faculty authored books will
be almost entirely excluded.
224
Penalizing schools and authors because
they publish books over articles is not desirable. Neither is discouraging
legal scholars from interdisciplinary work.
Hein recently added an integration with ORCID that allows legal
scholars to post items missing in Hein to ORCID and then push those
records to Hein.
225
However, it appears unlikely that U.S. News will use
222
Duplicate Content, MOZ, https://moz.com/learn/seo/duplicate-content
[https://perma.cc/DEL2-FPTW] (explaining how duplicate content leaves search algorithms
wondering which to rank, diluting the rank value of both items).
223
See, e.g., John R. Beatty, Citation Databases for Legal Scholarship, 39 LEGAL SERV. Q. 56,
74 (2020) (finding that Hein returned only 13 percent of the citations for a law professor who
published primarily in books and interdisciplinary journals).
224
See id. (finding that Hein returned less than 5 percent of the citations for a law professor
who mainly published in books).
225
Tara Kibler, HeinOnline Has Integrated with ORCID!, HEINONLINE BLOG (Jan. 27, 2020),
https://home.heinonline.org/blog/2020/01/heinonline-has-integrated-with-orcid/
[https://perma.cc/6LSS-FRX2].
234
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
this data for its initial ranking.
226
Hein is developing a tool to find
citations to works pulled in from ORCID within Hein’s set of full-text
journals
227
but Hein’s current system does not count citations to works
added via ORCID at all. Hopefully, once this new tool is complete, U.S.
News will find a fair and reliable way to incorporate this data into future
rankings. Still, as Bonnie Shucha has detailed, the process of integrating
ORCID and Hein is a time-consuming one that requires a significant
amount of manual work, which may delay or prevent some schools from
completing the process.
228
Further, Hein’s new tool will not be able to
“count citations to interdisciplinary scholarship and books in ORCID
from other interdisciplinary scholarship and books because those other
works are not part of Hein’s full-text collection.”
229
Unfortunately, we do not have a clear, easy to implement solution to
this issue. U.S. News could add data from other interdisciplinary
databases to what it receives from Hein. This would provide a more
inclusive picture. Still, we do not have a solution to track citations
within books.
Issues with OCR Text Recognition
In work on a different citation project, we found several errors in titles
exported from Hein. We also identified some situations where Hein’s
system treated each entry in a journal’s book review section as an
individual work and others where the book reviews were not separated
out. These discrepancies may lead to inaccuracies in the citation
computations. Further, John Beatty also found 648 citations to legal
journals that should have been in Hein but were not.
230
Beatty
hypothesizes that one possible reason could be issues with Hein’s
226
Shucha, supra note 23.
227
Id.
228
Id.
229
Id.
230
Beatty, supra note 223.
2021]
WILLEY & KNAPP
235
optical character recognition (OCR) process, especially considering that
these citations were found in Westlaw.
231
OCR also likely caused the
issues we observed. As with some of the other issues we identified, we
do not see a significant cause for alarm here. We anticipate that Hein,
or the underlying software it relies on, will continue to improve and that
eventually this will cease being an issue.
“No Follow”: Avoid Citations Counting When You Do Not Support
the Scholarship
In the SEO world linking to other websites benefits their rank in
Google’s algorithm. Sometimes website authors do not want other sites
to benefit from their linking to them — maybe the link is meant to show
a negative example. Google has built a system where authors can tag
links like this with “rel=‘no follow’.”
232
When website authors add this
tag, Google will generally not factor the link in their ranking algorithm.
Similar issues arise in legal scholarship. U.S. News and Hein should
create a method that will allow scholars to not have citations be credited
to the author they cite. Of course, as Google has, they will need to create
nuances in this rule to prevent abuse.
There are a host of other possible problems should U.S. News move
forward with their new ranking system. Likely, no one will be able to
identify and address all the issues until the system is launched. Perhaps
this is why U.S. News plans to roll it out as a separate ranking first: start
with a low-stakes scholarly ranking, work out the bugs over the course
of a few years, and then incorporate the metric into the overall rankings.
It is easy to find fault and poke holes in innovative ideas. Our goal with
this section is to raise potential issues early so that they can be addressed
231
Id.
232
Qualify Your Outbound Links to Google, GOOGLE DEVELOPERS: GOOGLE SEARCH CENT.,
https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links
[https://perma.cc/MR3R-ULD9].
236
THE OHIO STATE TECHNOLOGY LAW JOURNAL
[Vol. 18.1
and corrected quickly. The experience of SEO shows that it is possible
to greatly reduce bad behavior. We hope that U.S. News will address the
items we discussed and eventually incorporate the new scholarship
ranking into its overall law school ranking.
Conclusion
In this article, we focused on providing legal authors with methods that
may increase their citation counts. We also pointed out potential issues
for exploiting U.S. News’s proposed scholarship ranking. This may lead
some to ask if our suggested proposals are exploiting the system. There
is a difference between tailoring an article to meet the preferences of
readers and purposefully manipulating an article solely to increase its
citations.
233
We believe that whenever there is a gray area, authors
should think of their readers and would-be citers. Will the edit benefit
them? If so, do it. If not, ask whether the edit deceives. We discussed
several deceptive behaviors, like adding authors to an article when they
made no contribution or cramming keywords into footnotes in hopes of
a higher rank. To those practicing or considering deceptive behavior,
we close with a few words of caution. First, deceptive practices are
morally dubious. Second, it is likely that abusers will eventually be
found out and their schools may be punished as Google punished black
hat practices in the SEO world. In fact, abusers are already being
identified and reprimanded in other fields, both formally by
publishers
234
and informally by peers studying and drawing attention to
bad behavior.
235
233
van Wesel, Evaluation by Citation, supra note 197, at 203 (discussing a similar issue in the
scientific world).
234
See Phil Davis, Gaming the Impact Factor Puts Journal in Time-Out, SCHOLARLY KITCHEN
(Oct. 17, 2011), https://scholarlykitchen.sspnet.org/2011/10/17/gaming-the-impact-factor-
puts-journal-in-time-out/ [https://perma.cc/T73Y-D5HV] (discussing how Thompson Reuters
de-listed a journal from their Journal Citations Reports because the journal was intentionally
manipulating its citation numbers by citing more to articles in the journal).
235
See, e.g., Ioannidis, supra note 151, at 2-3 (finding evidence of excessive self-citation and
“citation farms” in a study of scientific authors); Errami & Garner, supra note 217 (finding
instances of bio medical researchers publishing the same results in multiple places).
2021]
WILLEY & KNAPP
237
We wish all our readers the best in their efforts to increase the visibility
of the valuable scholarship they are doing and look forward to U.S.
News’ efforts to better their ranking system.
