Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
1
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
Name of Program/Strategy: Reward & Reminder
Report Contents
1. Overview and description
2. Implementation considerations (if available)
3. Descriptive information
4. Outcomes
5. Cost effectiveness report (Washington State Institute of Public Policy – if available)
6. Washington State results (from Performance Based Prevention System (PBPS) – if available)
7. Who is using this program/strategy
8. Study populations
9. Quality of studies
10. Readiness for Dissemination
11. Costs (if available)
12.
Contacts for more information
______________________________________________________________________
1. Overview and description
Reward & Reminder, a population-level intervention targeting whole communities, counties, or States, is
designed to promote the community norm of not selling tobacco to minors. By using rapid and public
rewards and recognition for clerks and retailers/outlets that do not sell tobacco to minors, Reward &
Reminder aims to reduce illegal sales of tobacco, perceived access to tobacco, and tobacco use
prevalence rates. The intervention emerged from a contextual analysis of factors affecting the behavior of
store clerks, retailers, and the tobacco industry overall. At the core of the program is the use of "mystery
shoppers," teams of youth who--with parental permission and under the supervision of adults--enter
stores and try to buy tobacco products. They provide immediate recognition and rewards, such as gift
certificates, to clerks who do the "right thing" and give reminders to those who do not. The results of the
mystery shopper visits are entered into a Web-based system where they are made publicly visible, and
the results are communicated to local media to promote the positive norm. The mystery shopper visits are
scheduled across the year to maximize the immediate and sustained impact.
2. Implementation considerations (if available)
Using on-site or Web-based training, community adults who pass background checks can be trained as
supervisors in about 3 hours, and approved youth can be trained as shoppers in about an hour. Quality
control is maintained using a Web-based data entry system that facilitates data collection and guides
implementation and fidelity activities.
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
2
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
3. Descriptive information
Areas of Interest
Substance abuse prevention
Outcomes
1: Illegal sales of tobacco to minors
2: Illegal purchase of tobacco by minors
3: Tobacco use by minors
4: Social sources of tobacco for minors
Outcome Categories
Environmental change
Social functioning
Tobacco
Ages
13-17 (Adolescent)
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
Gender
Data were not reported/available.
Races/Ethnicities
American Indian or Alaska Native
Asian
Black or African American
Hispanic or Latino
Race/ethnicity unspecified
Settings
Other community settings
Geographic Locations
Urban
Suburban
Rural and/or frontier
Implementation History
Since Reward & Reminder was first implemented in 1989, it has been
used across three States (Kansas, Wisconsin, and Wyoming) and in
multiple communities in Oregon, reaching an estimated 9 million
community members. Four studies of the program have been conducted
and described in published articles.
NIH Funding/CER Studies
Partially/fully funded by National Institutes of Health: Yes
Evaluated in comparative effectiveness research studies: No
Adaptations
No population- or culture-specific adaptations were identified by the
applicant.
Adverse Effects
No adverse effects, concerns, or unintended consequences were
identified by the applicant.
IOM Prevention Categories
Universal
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
3
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
4. Outcomes
Outcome 1: Illegal sales of tobacco to minors
Description of Measures
Illegal sales of tobacco to minors were assessed by:
The percentage of tobacco outlets in the community that was
willing to sell tobacco to young "mystery shoppers," pairs of
youth who (with permission from their parents and under
supervision by adults) entered stores and attempted to buy
tobacco products.
Synar surveys, administered annually by States under the
supervision of SAMHSA's Center for Substance Abuse
Prevention. These surveys document direct attempts to buy
tobacco by underage youth. The percentage of outlets willing to
sell tobacco to young people over time was measured.
Key Findings
In one study, the proportion of outlets that were willing to sell
tobacco to minors was assessed in two communities once every 3
weeks prior to the intervention. After three baseline assessments,
Reward & Reminder was introduced into both communities.
Following the intervention phase, seven more assessments were
conducted, one every 3 weeks. Across the two communities, the
average percentage of outlets willing to sell tobacco to minors
across baseline assessments was 57%. This rate declined to 22%
across post-intervention assessments, representing a 62%
reduction in sales (p= .018).
In another study, the proportion of outlets that were willing to sell
tobacco to minors was assessed across several communities in
each of two States. The two States reported that outlets were willing
to sell tobacco to minors an average of 43% and 35% of the time
across baseline assessments. After implementation of Reward &
Reminder, these rates declined to an average of 8.1% and 10.8%,
respectively, across post-intervention assessments (p values < .01).
Studies Measuring Outcome
Study 1, Study 2
Study Designs
Quasi-experimental
Quality of Research Rating
3.3 (0.0-4.0 scale)
Outcome 2: Illegal purchase of tobacco by minors
Description of Measures
Illegal purchase of tobacco by minors was measured using the
Youth Risk Behavior Survey (YRBS). Youth reported on their direct
retail or vending machine purchases of tobacco during the past 30
days.
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
4
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
Key Findings
The proportion of minors illegally purchasing tobacco was assessed
across several communities in each of two States. Across baseline
assessments, an average of 13.3% and 11.4% of youth from the two
States reported buying tobacco from retailers or vending machines.
After implementation of Reward & Reminder, these rates decreased
to an average of 6.3% and 6.2%, respectively, across post-
intervention assessments (p values < .01).
Studies Measuring Outcome
Study 2
Study Designs
Quasi-experimental
Quality of Research Rating
3.3 (0.0-4.0 scale)
Outcome 3: Tobacco use by minors
Description of Measures
Tobacco use by minors was measured using the YRBS. Youth
reported on the frequency of their tobacco use during the past 30
days.
Key Findings
The proportion of minors using tobacco was assessed across
several communities in each of two States. Across baseline
assessments, an average of 37.1% and 33.3% of youth from the two
States reported smoking in the past 30 days. After implementation
of Reward & Reminder, these rates decreased to an average of
24.7% and 23.0%, respectively, across post-intervention
assessments (p values < .01). Similar results were found for the
percentage reporting daily smoking in the past month, which
decreased from an average of 14.5% and 13.3% to an average of
9.2% and 7.6%, respectively (p values < .01).
Studies Measuring Outcome
Study 2
Study Designs
Quasi-experimental
Quality of Research Rating
3.2 (0.0-4.0 scale)
Outcome 4: Social sources of tobacco for minors
Description of Measures
Social sources of tobacco for minors were assessed using the
YRBS. Youth reported on their access to cigarettes from social
sources (i.e., "gave someone else money to purchase," "borrowed
them from someone else," "person 18+ gave them to me") or from
stealing from a store or family member.
Key Findings
The proportion of minors obtaining tobacco from social sources was
assessed across several communities in each of two States. Across
baseline assessments, an average of 21.8% and 20.3% of youth
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
5
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
from the two States reported obtaining tobacco from social sources.
After implementation of Reward & Reminder, these rates decreased
to an average of 14.9% and 14.1%, respectively, across post-
intervention assessments (p values < .01). Similar results were
found for the percentage reporting stealing tobacco from a store or
family member, which decreased from an average of 3.6% and
2.5% to an average of 1.1% and 0.9%, respectively (p values < .01).
Studies Measuring Outcome
Study 2
Study Designs
Quasi-experimental
Quality of Research Rating
3.2 (0.0-4.0 scale)
5. Cost effectiveness report (Washington State Institute of Public Policy – if
available)
6. Washington State results (from Performance Based Prevention System (PBPS)
– if available)
7. Who is using this program/strategy
Washington Counties
Oregon Counties
8. Study populations
The studies reviewed for this intervention included the following populations, as reported by the study
authors.
Age
Gender
Race/Ethnicity
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
Data not
reported/available
65% Race/ethnicity
unspecified
19% American Indian
or Alaska Native
9% Hispanic or Latino
5% Asian
2% Black or African
American
13-17 (Adolescent)
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
Data not
reported/available
Data not
reported/available
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
6
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
9. Quality of studies
The documents below were reviewed for Quality of Research. Other materials may be available. For more
information, contact the developer(s).
Study 1
Biglan, A., Ary, D., Koehn, V., Levings, D., Smith, S., Wright, Z., et al. (1996). Mobilizing positive
reinforcement in communities to reduce youth access to tobacco. American Journal of Community
Psychology, 24(5), 625-638.
Biglan, A., Henderson, J., Humphrey, D., Yasui, M., Whisman, R., Black, C., et al. (1995). Mobilizing
positive reinforcement to reduce youth access to tobacco. Tobacco Control, 4, 42-48.
Study 2
Embry, D. D., Biglan, T., Hankins, M., Dahl, M. J., & Galloway, D. (2008). Evaluation Reward and
Reminder visits to reduce tobacco sales to, and tobacco use by, young people: A multiple-baseline
across two States. Manuscript in preparation.
Supplementary Materials
Lewis, R. K., Paine-Andrews, A., Fawcett, S. B., Francisco, V. T., Richter, K. P., Copple, B., et al. (1996).
Evaluating the effects of a community coalition's efforts to reduce illegal sales of alcohol and tobacco
products to minors. Journal of Community Health, 21(6), 429-436.
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's reported results
using six criteria:
1. Reliability of measures
2. Validity of measures
3. Intervention fidelity
4. Missing data and attrition
5. Potential confounding variables
6. Appropriateness of analysis
For more information about these criteria and the meaning of the ratings, see Quality of Research.
Outcome
Reliability
of
Measures
Validity of
Measures
Fidelity
Missing
Data/Attrition
Confounding
Variables
Data
Analysis
Overall
Rating
1: Illegal sales of
tobacco to minors
3.1
3.6
3.2
3.8
2.5
3.4
3.3
2: Illegal purchase of
tobacco by minors
3.3
3.3
3.3
3.8
2.5
3.5
3.3
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
7
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
3: Tobacco use by
minors
3.3
3.3
3.3
3.8
2.6
3.3
3.2
4: Social sources of
tobacco for minors
3.3
3.3
3.3
3.8
2.5
3.3
3.2
Study Strengths
The specific procedures used to increase the reliability and validity of the primary outcome measure (e.g.,
use of different pairs of youth assessors during the data collection process, inter-assessor reliability
checks) were well documented. Inter-assessor reliability was reported to be 100%. Manuals were
developed to guide implementation, and exposure to the specific intervention components was closely
tracked over the intervention period, demonstrating intervention fidelity across the studies reviewed and
supporting conclusions about the intervention's effects. One study used data from a Federal system that
is considered to be valid and reliable (i.e., the Synar survey) to measure the impact of the environmental
policy change.
Study Weaknesses
In both studies, use of the time series analysis limited the ability to control for some confounding
variables. Prevalence data from the YRBS were not collected as frequently as would have been ideal for
the study design (YRBS data were collected every 2 years, whereas illegal tobacco sales data were
collected annually), weakening causal inference for effects on prevalence.
10. Readiness for Dissemination
The documents below were reviewed for Readiness for Dissemination. Other materials may be available.
For more information, contact the developer(s).
Dissemination Materials
Embry, D. D., Biglan, A., Hankins, M., Dahl, M. J., & Galloway, D. (2009). Effectiveness trial using
Reward & Reminder visits to reduce tobacco sales to, and tobacco use by, young people: A multiple-
baseline across two States. Manuscript in preparation.
PAXIS Institute. (2003). PAXForms information management system manual. Tucson, AZ.
PAXIS Institute. (2005). Reward & Reminder: Training community youth and adults to make a difference
in tobacco use prevention. Tucson, AZ: Author.
PAXIS Institute. (n.d.). Dissemination protocols for Reward & Reminder. Tucson, AZ: Author. PAXIS
Institute online training videos (Licking County and Prevention First)
Program Web site, http://www.rewardandreminder.com
Sample customized materials from Kansas implementation:
Checklist for Implementation
Consultation, training, and technical assistance contract information
Guerilla Marketing Communication Tips
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
8
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
Kansas Media Plan
Memo about Synar results estimates
Memo for improving Synar outcomes
PAXIS Institute. (n.d.). Kansas Synar report. Tucson, AZ: Author.
PowerPoint training slides
R&R Kansas Action Plan
Training scripts and discussion points
TV Recommendation for Kansas Media Campaign for "It's Everybody's Business" Sample
customized materials from Wisconsin implementation:
Kick-off press release draft
PAXIS Institute. (2004, March). Wisconsin Synar Plan. Tucson, AZ: Author.
Sample ads for Wisconsin Wins
Thank You, Wisconsin [flyer]
Wisconsin Plan print flyer
WI Wins 2-page flyer
WI Wins interactive CD-ROM
WI Wins letter
WI Wins thank-you card
Youth Manual
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination using three
criteria:
1. Availability of implementation materials
2. Availability of training and support resources
3. Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for
Dissemination.
Training and Support
Resources
Quality Assurance
Procedures
Overall Rating
2.4
3.1
2.8
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
9
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
Dissemination Strengths
The examples provided of site-specific materials created for implementation are well developed and
include electronic versions for easy duplication. Extensive consultation is provided to new implementation
sites for the development and use of site-specific implementation materials and use of quality assurance
tools. Training materials provide a clear outline of the skills needed for implementation. An information
system developed specifically for this program is used for capturing information on implementation
activities to facilitate quality assurance.
Dissemination Weaknesses
Potential implementation sites can request developer consultation for a fee, but apart from this service,
sites are provided with little information about how to prepare for implementation, requirements for
staffing, the types of organizations best suited for implementation, scheduling of program activities, or
adaptations required for large-scale (i.e., statewide) versus small-scale (i.e., local community)
implementation. While both training and support are available, there is no uniform system for teaching
new sites how to implement this program. Little information is provided on how implementers should use
quality assurance data to improve program delivery.
11. Costs (if available)
The information below was provided by the developer and may have changed since the time of review.
For detailed information on implementation costs (e.g., staffing, space, equipment, materials shipping and
handling), contact the developer.
Item Description
Cost
Required by
Program
Developer
Hard copies of implementation
materials (print materials for
population of 20,000-90,000)
$1,865
Yes, one implementation
material option is required
Electronic materials (includes 2-
year materials license and
customization fee)
$1,000
Yes, one implementation
material option is required
On-site training
$1,500 plus travel expenses
Yes, one training option is
required
Web-based training
$1,500
Yes, one training option is
required
2 years of technical assistance and
consultation (includes license to
use trademark, phone/email/Web
technical support, access to
PAXForms, and Web site
setup/management)
$7,000
Yes
Reward & Reminder online data
management system
$300
No
Excellence in Prevention – descriptions of the prevention
programs and strategies with the greatest evidence of success
10
Excellence in Prevention is a project of Oregon Addiction and Mental Health Services and Washington
Division of Behavioral Health and Recovery. Information is drawn from many sources, including the
National Registry for Effective Prevention Programs (NREPP), sponsored by the Center for Substance
Abuse Prevention.
Additional Information
The costs of the license, materials, and services to be purchased from PAXIS Institute depend on the size
of the community, county, or State population and on the period of implementation. For example, the
estimated base price for a 2-year implementation with a population of 20,000 to 99,999 is $11,665.
Replications
Selected citations are presented below. An asterisk indicates that the document was reviewed for Quality
of Research.
Biglan, A., Ary, D., Koehn, V., Levings, D., Smith, S., Wright, Z., et al. (1996). Mobilizing positive
reinforcement in communities to reduce youth access to tobacco. American Journal of Community
Psychology, 24(5), 625-638.
Biglan, A., Henderson, J., Humphrey, D., Yasui, M., Whisman, R., Black, C., et al. (1995). Mobilizing
positive reinforcement to reduce youth access to tobacco. Tobacco Control, 4, 42-48.
Embry, D. D., Biglan, T., Hankins, M., Dahl, M. J., & Galloway, D. (2008). Evaluation Reward and
Reminder visits to reduce tobacco sales to, and tobacco use by, young people: A multiple-baseline
across two States. Manuscript in preparation.
Lewis, R. K., Paine-Andrews, A., Fawcett, S. B., Francisco, V. T., Richter, K. P., Copple, B., et al. (1996).
Evaluating the effects of a community coalition's efforts to reduce illegal sales of alcohol and tobacco
products to minors. Journal of Community Health, 21(6), 429-436.
12.
Contacts for more information
For information on implementation:
Miriam Willmann
(608) 772-0289
[email protected]
For information on research:
Dennis D. Embry, Ph.D.
(520) 299-6770
[email protected]
Learn More by Visiting: http://www.rewardandreminder.com