30 The Political Methodologist, vol. 20, no.2
rights, the seemingly random attrition may be even more
important for statistical inference in a panel study. Fig-
ure 1 shows the evolution of our panel’s demographics. Par-
ticularly in the first four waves, collected in a one-month
period around the Supreme Court decision (highlighted as
the event in the figure), our panel demographics remained
stable as people fell out of the panel. While there were some
deviations in the fifth wave, collected four months later, our
data suggest that panel attrition was ess entially more or
less equal across the categories of respondent traits, such as
race, gender, partisanship, and income. Interestingly, slight
trends are evident in age and education, with 18 and 19
year olds and those with some college education consistently
falling out slightly more often than the older and more edu-
cated, respectively. It is somewhat intuitive to expect that
across five months, the older and more educated would be
more reliable survey respondents. In all, however, we be-
lieve that the results of our study suggest that there is little
danger in the panel attrition in MTurk samples across the
typical demographics.
Leveraging the Cloud
Turkers respond to survey requests incredibly quickly, which
is especially valuable when conducting surveys intended to
explore how individuals respond to real events in an obser-
vational setting. Indeed, the pre-post event design has the
unique potential to c apture any micro-level change in this
context, which is one of several typical reasons to collect
panel data (see Sharot, 1991). Moreover, because events
in the real world unfold quickly and often change with the
discovery of new information, related events or media cov-
erage, it is important that the panels surrounding the event
are tightly situated around it. MTurk makes it possible to
launch a survey and collect hundreds of responses within
hours. Even in latter waves of our panel, we registered
dozens of responses in the first several minutes of posting
the HIT. In our case, this allowed us to end waves in antici-
pation of the health care decision and start a new one right
after it was announced. While we did not need to, it also
would have allowed us to quickly go back into the field with
another wave if, for example, President Obama had made
a health care speech a few days after an adverse decision.
Thus, along with MTurk’s low cost and ease of use, the
low panel attrition and quick response times of the MTurk
sample provide unusual flexibility for researchers to quickly
adapt their research design to real world events. In our
study of the ACA decision, scheduled weekly panels would
have been much less efficient than panels quickly and easily
adaptable to the event’s uncertain timing. Our case is not
unique; for example, one who is interested in public opinion
in response to natural disasters could time surveys around
forecasts and collect data in tight windows as events unfold.
A related benefit of doing MTurk panel studies around
events is the ability to do what we are calling real-time sur-
vey experiments, in which one randomizes participants to
realistic treatments related to the event as they are simul-
taneously getting political information about it in their nat-
ural environments. Relative to conducting artificial exper-
iments with captive audiences, this approach dramatically
increases external validity and offers a unique opportunity to
combine experimental treatments with the unfolding events
in the real world, thereby reconciling some of the known
discrepancies in survey and experimental studies (Hovland,
1959). Indeed, there has been some noteworthy attention
to the implications of captive audiences in political science
(e.g., Arceneaux and Johnson, 2010; Gaines and Kuklinski,
2011; Gaines, Kuklinski and Quirk, 2007; Levendusky, 2011)
and even to realistic experiments of attitude change in the
longitudinal context (Druckman, Fein and Leeper, 2012).
However, outside of using an MTurk sample with an online
survey platform or a convenience sample, combining a large
N panel design around imprecisely timed real world events
with realistic experimental treatments would likely be too
costly and/or would require sacrificing external validity due
to its advanced planning demands.
Our approach using MTurk allows one to not only cap-
ture the micro-level change around the event, but also to
leverage any effects of the event with a related experiment.
That is, we might be interested in not only attitude change
as a result of an event, like the ACA decision, but also with
factors associated with the event that can be experimentally
manipulated, such as media frames. In our case, we took ad-
vantage of news reports that Chief Justice Roberts switched
sides for political reasons that broke within days of the deci-
sions. While a typical experiment might expose participants
to this story, and this story only, weeks or months after a
decision to investigate the effect of a story about politicians
in robes, MTurk enabled us to expose some participants to
it as it was unfolding and as they were also self-selecting
other news and information about the decisions. An exam-
ple of a similar application would be studying the effects
of media frames leading up to and/or after presidential de-
bates. MTurk would allow a researcher to eas ily conduct
a pre-wave and then expose participants to different media
frames in the days after the debate to estimate their effect
on people who are also exposed to other uncontrolled infor-
mation.
To b e sure, our health care study is but one example of
using MTurk to conduct an inexpensive and flexible panel
study around a foreseeable political event. It enabled us
to ask dozens of questions of our choosing in each wave, to
adopt the timing of our surveys to events as they unfolded,
and to embed experiments by exposing participants to a
little-known yet important news story as it broke. Supreme
Court cases are but one pote ntial application. As we men-
tioned above, whether one wants to collect pre-post data
around precisely scheduled e vents, like presidential debates