Published on in Vol 7, No 2 (2021): February

Preprints (earlier versions) of this paper are available at, first published .
Evaluating the Effectiveness of a Web-Based Program (POP4Teens) to Prevent Prescription Opioid Misuse Among Adolescents: Randomized Controlled Trial

Evaluating the Effectiveness of a Web-Based Program (POP4Teens) to Prevent Prescription Opioid Misuse Among Adolescents: Randomized Controlled Trial

Evaluating the Effectiveness of a Web-Based Program (POP4Teens) to Prevent Prescription Opioid Misuse Among Adolescents: Randomized Controlled Trial

Original Paper

1Center for Technology and Behavioral Health, Geisel School of Medicine at Dartmouth College, Lebanon, NH, United States

2HealthSim Inc, Hanover, NH, United States

3Square2 Systems, Inc, Hanover, NH, United States

4Geisel School of Medicine at Dartmouth College, Hanover, NH, United States

5Department of Biomedical Data Science, Center for Technology and Behavioral Health, Geisel School of Medicine at Dartmouth College, Lebanon, NH, United States

Corresponding Author:

Sarah K Moore, MSW, PhD

Center for Technology and Behavioral Health

Geisel School of Medicine at Dartmouth College

46 Centerra Pkwy

Lebanon, NH, 03766

United States

Phone: 1 9176789280


Background: Prescription opioid (PO) use is common among adolescents in the United States. Despite recent declines from unprecedented peaks in adolescent PO use (eg, in 2012-2013), there is seemingly paradoxical evidence that PO-related consequences (eg, opioid use disorder and overdoses) are increasing. These trends and their possible consequences emphasize the importance of prevention efforts targeting PO misuse. To our knowledge, we have developed the first interactive web-based program (POP4Teens [P4T]) focused specifically on the prevention of PO misuse among adolescents.

Objective: This study aimed to evaluate the effectiveness of P4T, a web-based program designed to prevent adolescent PO misuse, in comparison with JustThinkTwice (JTT), an active control website, on PO-related attitudes, knowledge, risk perception, and intentions to use.

Methods: We conducted a web-based randomized controlled trial in 2018. A total of 406 adolescents (aged 12-17 years) were randomly assigned to either P4T or JTT. The outcome variables were attitudes, knowledge, and risk perceptions associated with PO misuse, intentions to use POs, and program feedback. Data were collected at baseline and at 1, 3, and 6 months.

Results: Both programs resulted in significant and sustained improvements in intention to use POs, increased perceived risk, impacted expectancies consistent with prevention, and improved PO refusal skills. P4T produced significantly greater increases in PO-related knowledge than JTT did, and it was reportedly easier to use and more liked. Baseline scores for youth reporting past-year medical use of POs, friends who engage in nonmedical use of POs, and/or poor mental health underscored their at-risk status compared with youth from the other groups.

Conclusions: P4T positively impacted all study variables that are known to prevent PO misuse among teens. Moreover, its web-based nature simplifies the dissemination and implementation of this novel tool designed to help meet the challenges of the evolving national opioid crisis.

Trial Registration: NCT02737696;

JMIR Public Health Surveill 2021;7(2):e18487




Prescription opioid (PO) use is common among adolescents in the United States. Over the last 4 decades, the lifetime prevalence of medical (13%-20%) and nonmedical use of POs (NMUPO; 7%-13%) among high school seniors (approximately 17-18 years of age) has fluctuated yet both remain high [1]. One in every 100 adolescents aged between 12 and 17 years reported current NMUPO in 2017 [2], and this estimate may underreport the problem due to potential social desirability bias associated with self-report measures, particularly among youth [3]. Predictors of NMUPO and opioid use disorder (OUD) include early initiation of medical opioid use, and the risks of OUD are inversely correlated with age of onset [4-6]. Adolescents with mental health problems are also more likely than those without such problems to use diverted POs [7]. Adolescents with NMUPO overwhelmingly report accessing these medications from friends and family and having friends or family who are prescribed opioids increases the risk for misuse and OUD [8-10].

Compared with nonusers, adolescents who engage in NMUPO have increased odds of using alcohol and other drugs, engaging in criminal behaviors, using health care (eg, receiving emergency or inpatient treatment), receiving treatment services for mental health, and reporting poorer health [11-15]. The risk of transition to heroin use is greatest among those that begin NMUPO early in adolescence [16-19], and fatal and nonfatal overdose rates are higher for these youth compared with their peers who do not engage in NMUPO [20]. Despite recent declines from unprecedented peaks in adolescent PO use (eg, in 2012-2013), there is seemingly paradoxical evidence that PO-related consequences (eg, OUD, mortality, overdoses) are increasing [21,22]. These increases are likely associated with evidence that youth with OUD rarely receive the treatment they need [23].

These trends and their possible consequences emphasize the importance of prevention efforts targeting PO misuse. To our knowledge, we have developed the first interactive web-based program (POP4Teens [P4T]) focused specifically on the prevention of PO misuse among adolescents. This program extends our previous work to develop and evaluate science-based, computer-delivered substance misuse prevention programs for youth, including a prototype of this program we previously developed and evaluated [24-26]. The scientific evidence underpinning P4T’s program content comes from research on identified risk factors for PO use among youth [27], computer-delivered interventions [28], and computer-assisted instruction technology [29].

The P4T program is unique compared with other programs, as it is the first of its kind to incorporate knowledge about risk factors for PO misuse. PO misuse is unique compared with misuse of many other drugs: the threshold to access is low, potentiating the behavior (ie, usually obtained from friends and family and at no cost) [30,31]. Relatedly, youth who are prescribed opioids often divert their medication [32,33]; the perceived risks of misuse are significantly attenuated compared with other drugs, as they are dispensed by trusted medical providers [34,35]; youth perceive that parental consequences of misuse will be minor if they are caught with POs relative to other drugs [36].

In addition to a focus on the specific risk factors associated with POs, the interactive, activity-oriented P4T program is rooted in the prevention science literature, which has demonstrated that effective substance abuse prevention programs educate youth about the mechanism of action by which a substance produces its pharmacological effect and additionally include various types of skills training (eg, to resist peer, family, media, and other social influences known to promote drug use, and/or general skills training around social competency and coping with stressful life situations) [37,38]. Skills training increases protective factors, reduces risk factors, and produces reductions in drug use among youth [28]. Notably, programs based on social influence and general skills training models have been shown to be effective when delivered via interactive, activity-oriented programs and not traditional, didactic instructional techniques [39].

The P4T program is also unique as it is web-delivered and mobile-friendly and therefore available for use in a variety of settings. A recent scoping review of consumer health information technology (CHIT) in the prevention of substance use [40] underscores the effectiveness of user-centered, interactive web-mediated apps designed to improve information access and exchange, decision making, and facilitate behavior changes that promote health and well-being. Traditional means of reaching youth are hamstrung by costs associated with teacher or clinician-delivered interventions as well as manifold issues impeding intervention delivery fidelity. As of 2018, 94% of teens reported going on web daily and almost 95% reported mobile device usage and/or ownership [41]. Compared with traditional delivery methods, CHIT interventions delivered via digital platforms boast benefits of greater appeal to youth [28], reduced costs [36], standardized content delivery [36], potential for delivery across different settings (eg, on the web, home, community organizations, and health care providers’ offices) [37], and minimized training and/or clinical staff burden [42].

Published scientific literature on digital platforms for interventions targeting adolescent substance misuse is burgeoning [43,44]. Such programs have demonstrated effectiveness in 3 contexts: primary care [45], schools [44], and homes [46-48]. They target the prevention of different classes of drugs: alcohol [43,44], cannabis [45,49], cigarettes [50] and generally consist of interactive web-based activities that function to increase drug-related knowledge and shape user attitudes and normative beliefs about substance use in ways that promote abstinence or delayed onset of use [51]. The web-based context for the P4T program focuses on a new class of drugs (POs) previously unexplored in a digital format and additionally employs unique informational technologies that are effective in promoting relevant knowledge and skills. The P4T content uses information technologies to teach content to mastery via interactive and computer-assisted instruction (engaged when youth access quizzes) and enables exploration of the information and skills being taught in a video-based simulated environment [29,52,53].


The randomized controlled trial described herein focused on the prevention of PO misuse among youth. This web-based trial is designed to evaluate the effectiveness of the web-based PO misuse prevention program P4T, developed by our research group, compared with an active control website JustThinkTwice (Drug Enforcement Administration [DEA]) to impact attitudes about knowledge and perceptions of risk associated with misuse of POs as well as intentions to use and actual use of POs. Details of the development and formative evaluation of P4T have been published elsewhere [24].


Participants were assigned at random using a nonlinear random number generator based on an AES block cipher in counter mode to either the experimental, web-based P4T program or the DEA’s JustThinkTwice (JTT) website (the active control condition). The study team was not blinded to randomization. However, due to the web-based nature of the trial (eg, recruitment, randomization, data collection) and limited-to-nonexistent research team contact with participants following randomization, bias was minimized.

The CONSORT (Consolidated Standards of Reporting Trials) checklist has been provided in Multimedia Appendix 1.


Adolescent participants (12-17 years old) were recruited for study participation through Facebook and Google AdWords. This age range was selected to capture youth before a notable increase in risk of hazardous consequences of nonmedical PO use observed from age 12 years to 14 years [6,54], through the end of high school or before college. To be eligible, youth needed to be willing to use the study website to complete assessments and participate in the interventions, have access to a computer, a tablet, or a mobile device with an internet connection, and be able to hear audio.

From a user perspective, once an interested youth clicked on an ad, they were directed to the study landing page, which provided basic information about the study aims and clickable icons to learn more. Upon clicking on learn more, interested youth were offered the opportunity to receive a call from a research assistant by entering basic contact information (ie, name, email, parent’s name, email, phone number, and best time to be reached) that the research assistant used to make contact, answer additional questions, collect zip code data, email the consent form, and enable backend, automatic random assignment to a study condition. Once consent was established, the research assistant emailed the baseline assessment, which, once completed (as evidenced by a timestamp record on the program’s dashboard menu of participant activities), prompted the research assistant to send the first electronic gift card via email to the participant as compensation for their time spent in completing the assessment. Youth received US $20 for baseline, US $30 for 1-, 3-, and 6-month follow-ups and US $30 as bonus if all assessments were completed. Participants completed all assessments on the web in a secure manner, which allowed for greater anonymity and has been shown to result in more valid data compared with data collected via in-person interviews [55,56]. Delivering assessments via computer also ensured that questionnaire administration was more consistent, data processing was eliminated, and lag time between data collection and analyses was reduced. All procedures were approved by the Dartmouth College Committee for the Protection of Human Subjects.


The experimental program P4T is a newly developed intervention [24] designed to increase knowledge about POs and their risks and address social influences on PO use, such as developing skills to refuse offers or requests for POs. The program comprised 8 primary modules, and the information and skills emphasized in each module are delivered through 4 components: a storytelling component, an information lesson component, a skill-based video, and a quiz. The 8 module topics are (1) what are opioids? (2) misconceptions that opioids are safe and nonaddictive, (3) misconceptions that use of POs without a prescription is legal, (4) risks of PO misuse, (5) nonmedication alternatives for pain management, (6) refusing offers to misuse POs, (7) refusing requests by others for a PO prescribed to you, and (8) how to know if you or someone you may know may be addicted.

In each module, users were introduced to a different character based on real stories of youth in treatment for OUD. By clicking on the character icon, users were brought to an introductory screen displaying an exemplary quote from the youth’s story regarding PO use. Here, users could choose to listen to a recording of the story, read the story transcript, or view story highlights.

The program then allowed users to move between the introductory storytelling screen and the remaining 3 components (lesson, video, and quiz) personalizing the navigational flow and contributing to the interactive nature of P4T. The lessons detailed the scientific underpinnings of the story and delivered information through an interactive carousel format. The videos provided participants with concrete methods to approach social interactions and dilemmas involving PO misuse. The videos employed an action-reaction model in which a behavioral scenario was shown, followed by one or more characters reflecting on the situation. Finally, the quizzes allowed program users to test their understanding of the module content. Youth were given a variable amount of time to respond to quiz questions, which are presented multiple times and worded slightly differently to increase fluency and mastery, depending on how fluent youth are with the material, for example, if they are repeatedly getting a question incorrect, they will be allowed more time to respond to the next time they see a version of the question. Thus, the quizzes further contributed to the interactive nature of the program. Participants were asked to complete the 8 modules over the course of 4 weeks (approximately 2 modules per week).

The active, educational control website JTT is a web-based educational initiative supported by the DEA, which provides science-based information about teen drug abuse prevention. The intended purpose of the website, according to a DEA press release on the occasion of the launch of the teen website reads that it is “a website especially for our teenagers that provides young people with straightforward information on the consequences of drug use and trafficking, including health, social, legal, and international consequences. The site does not inundate the user with DEA messages; rather the facts come straight from leading physicians, scientists, and legal experts.” [57]

The website is not strictly dedicated to POs, although opioids are well represented. People navigating the website will find drug information, news and media, true stories of youth who have suffered due to drug use, information about consequences of drug use, drug use facts and statistics, videos, and a brief quiz. Randomized youth were asked to use this website for 30 min twice per week for 4 weeks.

Outcome Measures

Behavioral assessments accessible by links emailed to participants were completed at baseline and follow-up time points (1, 3, and 6 months). At baseline, participants completed a form measuring basic demographic and substance use history, for example, age, gender, race, ethnicity, grade in school, average school grades, lifetime prescriptions for opioids, past-year mental health status, substance use in general, PO use among participants and their friends, and past participation in PO abuse prevention programs. At the 1-month time point only, participants completed the program feedback survey. Given the pervasive misconceptions that many youths have in their understanding of the risks associated with PO misuse [58], we primarily focused on analyses of knowledge about PO misuse as well as measures that are highly predictive of future use (attitudes, intentions, skills, and perceived risk).

Attitudes Toward PO Misuse

Participants were asked about their expectancies associated with PO misuse. Positive expectancies (eg, reduce boredom) and negative expectancies (eg, get in trouble with parents) about such use were evaluated using a 5-point measure ranging from strongly disagree to strongly agree. Mounting evidence has suggested that outcome expectancies (expected or anticipated behavioral, cognitive, and emotional harms and benefits [59]) associated with substance use play a role in the initiation and progression of drug use [60]. We modeled these questions after expectancy items developed by researchers examining potential physical, social, and psychological consequence expectancies associated with alcohol consumption among youth [61]. Changes in expectancies appear to be an important target in substance use prevention across alcohol, tobacco, and marijuana [62].

Perceived Risk of PO Misuse

Participants were asked 2 separate questions about how much they think they risk harming themselves (1) physically and (2) in other ways if they try POs for nonmedical purposes (eg, 5-point scale ranging from no risk to great risk) [59]. These questions are used annually in the Monitoring the Future Study [63], and the reliability and validity of survey questions is extensively discussed in an occasional paper detailing design and procedures [64]. Other studies have also included questions modeled after survey questions [65]. Research has consistently shown that alcohol and cannabis use is less likely among youth who perceive those substances and/or substance use behaviors as risky for the user [66,67], and thus, perceived harmfulness is potentially central to early prevention strategies [68].

Knowledge About PO Abuse Prevention

Participants were asked to complete a 15-item, multiple-choice measure assessing knowledge about effective PO abuse prevention (eg, how opioids work in the body). Knowledge test questions were generated by the research team based on scientific literature associated with the 8 module topics in the P4T program (eg, nonmedication alternatives for pain management).

Skill Acquisition

Youth were asked to pretend a friend offered them a PO and estimate how hard it would be to refuse the offer (5-point scale of very easy to very hard) and their ability to refuse the offer (5-point scale of definitely would to definitely would not). The same format was used to ask youth to pretend a friend asked them to share or sell POs prescribed to them [69]. These items were modeled after drug resistance scales used in previous studies with youth of the same age [70,71].

Assessment of Usage Data

We quantified usage statistics of the web-based program by assessing the extent of module completion.

Behavioral Intentions to Use Opioids Without a Prescription

Youth were asked about their intentions to use opioids without a prescription even once or twice over the next 12 months using a 5-point scale ranging from definitely not to definitely will. Those who provided an affirmative response indicating any likelihood were also asked about the likelihood of using opioids without a prescription nearly every month for the next 12 months. This 2-item composite (α=.73) was modeled after a measure used in a study [72] that found changes in marijuana expectations among a large cohort of adolescents associated with subsequent future marijuana initiations.

Feedback Survey

The feedback survey was a brief 15-item tool that included 11 visual analog scale items (Multimedia Appendix 2) and 4 open-ended response items (ie, What youth liked best? Least? Suggestions? Anything else to add?). Possible values for visual analog scale scores ranged from 0 to 100 mm and were anchored by the variable of interest probed in the item (eg, how interesting was the section of the program you just completed? score range: 0=not interesting to 100=very interesting).

Data Analysis

Although random assignment was designed to account for baseline differences, groups were compared for differences in demographics and other characteristics using two-tailed t tests, Wilcoxon rank sum tests for nonnormal variables, and chi-square tests for categorical measures. Program feedback (collected at 1 month) was also compared via t tests. Primary analyses included all participants randomized to a study condition independent of early dropout, consistent with an intent-to-treat approach to clinical trials [73]. Analyses were performed using SAS statistical software (type I error=.05). Outcome measures were evaluated in separate analyses. Data from primary outcomes were evaluated using mixed effect models, which allow for nonindependent data within individuals and are robust to incomplete data. All participants with any baseline or follow-up assessments were included in the outcome analyses. In addition to the study condition indicator, these models included, in the fixed effects portion of the model, a main effect for time as well as a 2-way interaction between study condition and time. Before evaluating the comparative effectiveness of P4T, each program’s effectiveness in impacting primary outcome variables without comparison is reviewed, as neither has previously been evaluated. The key estimate from this model is the 2-way interaction effect, which indicates whether changes in outcome over time are different for the 2 conditions.

Sample Size Calculation

In our previous work [74], a computer-based prevention intervention, relative to an efficacious life skills control condition, had a large impact on knowledge (Cohen d=0.84), a moderate impact on drug refusal (Cohen d=0.43), a small impact on drug use intentions and attitudes (Cohen d=0.20), and a moderate impact on alcohol use (Cohen d=0.37). Schwinn et al [69] found a comparable effect size for reducing substance use when comparing an internet-based prevention program with an assessment-only control group (Cohen d=0.40). The planned sample size of 400 participants was based on having sufficient power to detect group differences in all outcomes, assuming 80% retention at the 6-month follow-up [73,75,76]. Assuming baseline assessments account for at least 9% of the variance in corresponding outcomes (r=0.30), power is estimated to be greater than 99% using α=.05 (2 sided) for detecting a prevention group effect of Cohen d=0.84 on the outcome measure of knowledge. Power was estimated to be 92% using α=.05 (2 sided) for detecting a prevention group effect of Cohen d=0.37 on the outcome measure of substance use and 98% using α=.05 (2-sided) for detecting a prevention group effect of Cohen d=0.43 on the outcome measure of drug refusal. For the intention to use outcomes, we determined the sample size needed to detect Cohen d=0.30, which is slightly higher than the effect observed in previous work [74], to reflect the use of an active control group in the planned study instead of a comparison condition of demonstrated effectiveness. The power to detect that effect was 80% when the sample size was 320 (after 20% attrition) at the 6-month follow-up. 


A nationwide sample of youth from 41 of the 50 US states was recruited between June 2017 and February 2018, and data collection was completed in September 2018. Of the 1470 youth and/or parent/guardian who signed up to gather more information about participation, 406 were consented and randomly assigned to one of the 2 websites (Figures 1 and 2).

Figure 1. CONSORT (Consolidated Standards of Reporting Trials) diagram.
View this figure
Figure 2. Geographic locations of participants. Zip code data for 379 of 406 participants (missing zip code data: n=27). In total, 41 of 50 states had at least one participant. Numbers in red circles are those of participants in a geographic area and may represent more than one state’s data. Alaska (n=4) is not shown on the map.
View this figure

Participant Characteristics

The sample was primarily female (285/405, 70.4%), with a mean age of 15.8 (SD 1.2) years, and high achieving with 70.9% (287/405) reporting A’s for average school grade; 92.8% (376/405) reported good-to-excellent physical health, and approximately 73.6% (298/405) reported good-to-excellent mental health. More than a quarter of the sample reported lifetime exposure to medically prescribed opioids. Randomization was largely successful, as groups did not differ significantly on any baseline variable except for race: Asian (P4T/JTT: 21%/13.5%); Black/African American (P4T/JTT: 5.9%/12%); and White (P4T/JTT: 64%/67.5%; P=.03). For additional participant characteristics, see Table 1.

Table 1. Demographic and baseline characteristics.
Demographic and baseline characteristicsOverall (N=405)P4Ta (n=205)JTTb (n=200)P value
Age (years), mean (SE)15.8 (1.2)15.7 (1.2)15.8 (1.2).80
Gender, n (%)


Female285 (70.4)148 (72.2)137 (68.5)

Male113 (27.9)54 (26.3)59 (29.5)

Transgender5 (1.2)1 (0.5)4 (2)

No answer2 (0.5)2 (0.9)0 (0)
Race, n (%)


American Indian or Alaskan Native3 (0.7)1 (0.5)2 (1)

Asian70 (17.3)43 (20.9)27 (13.5)

Black or African American36 (8.8)12 (5.9)24 (12)

Pacific Islander4 (0.9)4 (1.9)0 (0)

White266 (65.7)131 (63.9)135 (67.5)

Other26 (6.4)14 (6.8)12 (6)
Ethnicity, n (%)


Latino or Hispanic49 (12.1)29 (14.1)20 (10)

Non-Latino or non-Hispanic356 (87.9)176 (85.9)180 (90)
Grade, n (%)


6th-9th (ages 13-15 years)64 (15.8)31 (15.1)33 (16.5)

10th-12th (ages 16-18 years)324 (80)169 (82.4)155 (77.5)

Not in school or other17 (4.2)5 (2.4)12 (6)
Average school grade, n (%)


A (90-100)287 (70.9)144 (70.2)143 (71.5)

B (80-89)100 (24.7)52 (25.4)48 (24)

C (70-79)15 (3.7)7 (3.4)8 (4)

D (60-69)2 (0.5)1 (0.5)1 (0.5)

F (<60)1 (0.2)1 (0.5)0 (0)
Poor or fair mental health, n (%)


No298 (73.6)155 (75.6)143 (71.5)

Yes107 (26.4)50 (24.4)57 (28.5)
Physical health, n (%)


Excellent93 (22.9)50 (24.4)43 (21.5)

Very good177 (43.7)85 (41.5)92 (46)

Good106 (26.2)58 (28.3)48 (24)

Fair24 (5.9)8 (3.9)16 (8)

Poor5 (1.2)4 (1.9)1 (0.5)
Ever prescribed opioids, n (%)


Yes106 (26.2)50 (24.4)56 (28)

No299 (73.8)155 (75.6)144 (72)
Friends who use prescription opioids, n (%)


Yes68 (16.8)37 (18)31 (15.5)

No337 (83.2)168 (81.9)169 (84.5)
Participated in a prescription opioid prevention program, n (%)


Yes11 (2.7)4 (1.9)7 (3.5)

No394 (97.3)201 (98)193 (96.5)

aP4T: POP4Teens.

bJTT: JustThinkTwice.

Program Effectiveness (Within-Condition Effects)

Drug Use Outcome Expectancies (Attitudes)

The P4T program produced significant improvements from baseline across the follow-up time points (1, 3, and 6 months) for all positive expectancies of POs. The JTT program produced a similar pattern of results except for reduce boredom (baseline-1 month: P=.11) and weight loss (baseline-6 months: P=.06). The P4T program produced significant improvements from baseline across all follow-up time points for most negative expectancies. The exceptions were a pattern of nonsignificant results across all time points for get in trouble with parents and nonsignificance of feel sleepy (baseline-1 month: P=.41). The JTT program produced significant improvements from baseline across all follow-up time points for a few negative expectancies (eg, feel sleepy or overdose). The exceptions were a pattern of significant differences observed at the 1 and 6 month follow-ups, and nonsignificant differences at the 3-month follow-up (eg, spend too much money, feel sick, or pass out). The same pattern of nonsignificant differences across all time points for get in trouble with parents held for the JTT program. Finally, there were significant differences from baseline to 1 month that were not sustained at subsequent follow-ups for do poorly in school. For greater detail on within-condition effects, see Multimedia Appendix 2 (means) and Multimedia Appendix 3 (mixed models).

Perceived Risks

The P4T program significantly increased perceived physical risks from baseline to 1 month and baseline to 3 months; however, significant increases were not maintained at the third follow-up (P=.14). The program also significantly increased perceived other risks from baseline to 3 months; however, this effect was not observed for baseline to 1 month (P=.06) or baseline to 6 months (P=.06). The JTT program produced significant and sustained increases in perceived physical and other risks from baseline across all time points.


The P4T program significantly increased knowledge about POs from baseline to all time points. The JTT program produced no significant changes in knowledge at any follow-up time point.


The P4T program produced significant and sustained changes in the desired direction across all skill sets (ie, assessments of difficulty and ability to refuse offers and requests) and time points except for ability to refuse an offer to misuse POs at 3 months (P=.06). The JTT program produced significant and sustained changes in the desired direction across all skill sets except for perceived difficulty to refuse an offer to misuse POs at 1 month (P=.14).


Both programs produced significant and sustained decreased intentions to use POs in the next 12 months across all time points.

Assessment of Usage

Participants in the P4T group completed an average of 6.43 (SD 2.98) of 8 possible modules and a total of 80.79% (1312/1624) possible modules. We were unable to calculate a similar metric in the control sample (JTT), as we did not control that website. However, we were able to calculate the average number of log-ins on the control site (a conservative estimate as participants may have chosen to access the site without using the portal on the study website [mean 4.06, SD 3.89]).

Effectiveness of P4T Compared With JTT

Drug Use Outcome Expectancies (Attitudes)

There were no significant differences between groups for any positive expectancies at any time points. There were no significant differences between groups across the negative expectancies observed from baseline to 1 month. From baseline to 3 months, there were statistically significant increases in negative expectancies of POs favoring P4T: do poorly in school (P=.01), spend money (P=.01), feel sick (P=.03), and pass out (P=.004). From baseline to 6 months, the effect for do poorly in school was maintained (P=.01).

Perceived Risks

P4T produced a greater reduction (from baseline to 1 month) in the odds of perceiving a low other risk from POs compared with JTT (P=.03). There were no other significant differences between groups at any other time point for measures of risk.


There was a significant difference between groups in change from baseline for knowledge gains favoring P4T at the first 2 follow-up time points: baseline to 1 month (P<.001) and baseline to 3 months (P=.001). The effect somewhat diminished at the 6-month follow-up (P=.06).


There were no significant differences between groups across skills except for difficulty refusing an offer to misuse between baseline and 1 month (P=.05) favoring P4T. Differences were not maintained at later time points.


There were no significant differences between groups in change from baseline to any of the 3 follow-up time points for intention to use POs: baseline to 1 month (P=.68), baseline to 3 months (P=.99), and baseline to 6 months (P=.75).

Feedback Surveys

With one exception, evaluations of the P4T and JTT programs were generally positive, with most mean scores at 7 or higher (scale 1-10) and not significantly different. However, the P4T program was rated as significantly easier to use than the JTT program (P<.001). In addition, the P4T youth liked using laptops, phones, and/or tablets significantly more to access the program than JTT youth (P<.001). No mean scores fell below 6, except for the applicability of program content to participants’ lives (Multimedia Appendices 2 [means] and 3 [mixed models]).

Four P4T program dimensions that youth liked best (in order of preference) were stories, ease of use, quizzes, and video. The stories were preferred for how they provided real-life, relatable examples and helped make the opioid epidemic just seem more real. Youth liked P4T because of how easy it was to access and navigate. They also reported enjoying the quizzes for how the material began to make sense, and the repetitive nature of the quiz questions helped youth master the information. Finally, some youth appreciated that the videos were upfront with being a little bit corny at times and were a great educational tool. The functionality of quizzes and videos surfaced among program dimensions was the least. Specifically, the timed and repetitive nature of the quizzes surfaced for critique:

The timed portion of the quizzes created unnecessary stress, increasing speed of the questions…lead to testing for your reaction time rather than processing information, the questions seemed very repetitive.

Videos were least liked by some because they were found to be dull.

Baseline Differences: Characterizing the Sample in Terms of At-Risk Subgroups

Drug Use Outcome Expectancies (Attitudes)

Scores for youth reporting past-year medical use of POs, having friends who engage in NMUPO, and having poor mental health illuminated their at-risk status at baseline compared with youth who were not in those groups and therefore warrant attention. For negative expectancies, those with poor mental health agreed more strongly than those without poor mental health that overdose is likely when one engages in NMUPO. There was a pattern across those with a history of past year medical use of POs and having friends who engage in NMUPO that NMUPO would likely result in trouble with parents and doing poorly in school compared with those without such histories. In addition, those with a history of past year medical use of POs agreed more strongly that NMUPO would make one feel sleepy. For positive expectancies from POs, there is a clear pattern for both those with histories of poor mental health and having friends who engage in NMUPO demonstrating greater agreement with almost all such items compared with those without these histories, indicating an increased risk for these groups of youth. The only positive expectancy score for those with past year medical use of POs histories higher than their comparators was for reducing physical pain.

Perceived Risks, Knowledge, Skills, and Intentions

There were no differences between groups at risk (poor mental health, past year medical use of POs, and having friends who engage in NMUPO) and those not at risk in these categories for perceived risks with the lone exception among those with friends who engage in NMUPO who demonstrated slightly weaker agreement than those without friends who engage in NMUPO that there may be other risks (beyond physical), if one NMUPO. There were also no differences between youth in these 3 at-risk groups and their counterparts without such histories on knowledge about POs. Similar to the pattern of findings for positive expectancies, those with poor mental health and friends who engage in NMUPO scored at greater risk with respect to skills compared with participants without such histories. There were no differences between those with past year medical use of POs and those without such histories. Finally, all 3 subgroups reported stronger intentions to NMUPO in the next 12 months compared with youth not in those groups (Table 2).

Table 2. Primary and secondary measures of at-risk youth at baseline (mean).
Baseline valueaPoor mental healthMedical prescription opioid useFriend who NMUPOb

Yes, mean (SD)No, mean (SD)P valueYes, mean (SD)No, mean (SD)P valueYes, mean (SD)No, mean (SD)P value
Intention to use prescription opioids1.63 (0.77)1.43 (0.69).021.62 (0.76)1.44 (0.70).031.76 (0.92)1.43 (0.66).01
Negative expectancies

Spend too much (US $)3.92 (1.07)3.81 (1.11).413.73 (1.18)3.88 (1.07).233.85 (1.20)3.84 (1.08).93

Overdose3.09 (1.13)2.77 (1.11).012.87 (1.16)2.86 (1.11).922.79 (1.10)2.87 (1.13).60

Trouble with parents4.58 (0.82)4.53 (0.95).584.37 (0.99)4.60 (0.88).034.19 (1.14)4.61 (0.85).01

Pass out3.44 (0.93)3.30 (0.93).183.22 (0.98)3.38 (0.92).153.24 (0.95)3.35 (0.93).35

Do poorly in school3.93 (0.99)3.96 (1.09).793.70 (1.11)4.04 (1.03).013.60 (1.15)4.02 (1.03).01

Feel sick3.79 (0.94)3.80 (0.90).953.80 (0.97)3.80 (0.88).983.75 (1.00)3.81 (0.89).65

Feel sleepy3.85 (0.89)3.75 (0.90).343.95 (0.96)3.72 (0.87).033.79 (0.99)3.78 (0.88).89
Positive expectancies

Reduce anxiety3.12 (1.17)2.73 (1.05).0032.86 (1.12)2.82 (1.08).773.22 (1.08)2.75 (1.08).003

Reduce boredom2.45 (1.14)2.10 (1.07).012.08 (1.11)2.23 (1.10).232.49 (1.23)2.14 (1.07).03

Escape problems2.35 (1.10)2.01 (1.01).012.18 (1.07)2.07 (1.03).352.44 (1.15)2.03 (1.01).01

Feel good2.95 (1.04)2.56 (1.07).0012.84 (1.14)2.60 (1.04).063.03 (1.15)2.59 (1.04).004

Reduce physical pain3.67 (1.10)3.48 (1.04).123.81 (0.94)3.43 (1.08).0013.78 (1.08)3.48 (1.05).04

Reduce sadness2.76 (1.04)2.44 (1.00).012.42 (0.99)2.56 (1.03).242.76 (1.01)2.47 (1.02).03

Improve social situations2.31 (0.96)1.93 (0.93)<.0012.02 (1.02)2.03 (0.93).922.25 (1.06)1.98 (0.92).06

Lose weight2.57 (1.00)2.45 (0.99).312.46 (1.05)2.49 (0.97).792.51 (1.04)2.48 (0.98).80
Perceived risks

Physical4.28 (0.87)4.37 (0.81).334.30 (0.95)4.37 (0.78).534.22 (0.91)4.38 (0.81).20

Other4.35 (0.78)4.44 (0.76).284.40 (0.82)4.42 (0.74).774.21 (0.94)4.46 (0.72).04

How hard to refuse offer1.81 (0.95)1.49 (0.77).0021.58 (0.79)1.58 (0.85).961.79 (0.96)1.54 (0.80).04

Able to refuse offer1.69 (0.78)1.38 (0.72)<.0011.50 (0.88)1.45 (0.70).591.69 (0.87)1.42 (0.71).02

How hard to refuse request for prescription1.88 (1.05)1.52 (0.82).0021.58 (0.85)1.62 (0.92).621.93 (1.06)1.55 (0.86).01

Able to refuse request for prescription1.67 (0.89)1.46 (0.80).031.60 (0.97)1.49 (0.77).281.72 (0.96)1.48 (0.80).06
Knowledge13.31 (2.00)13.18 (2.18).5813.32 (2.15)13.18 (2.12).5613.28 (2.30)13.20 (2.09).80

aLikert item anchors or values: expectancies: strongly disagree=1, disagree=2, not sure=3, agree=4, strongly agree=5; perceived risks: no risk=1, not sure=3, great risk=5; skills or difficulty: very easy=1, easy=2, not sure=3, hard=4, very hard=5; skills or ability: definitely would=1, would=2, not sure=3, would not=4, definitely would not=5; intentions: definitely will not=1, will not=2, not sure=3, will=4, definitely will=5.

bNMUPO: nonmedical use of prescription opioids.

Principal Findings

To our knowledge, we developed the first interactive, web-based, and mobile-friendly program focused specifically on the prevention of PO misuse among adolescents. This study is the first to evaluate the effectiveness of this novel tool designed to help meet the challenges of the evolving opioid crisis in the United States.

The study yielded 4 key findings. First, the results demonstrate that both the P4T and the comparison JTT programs performed well in targeting variables known to impact NMUPO. Both programs were effective in steadily and significantly decreasing agreement with positive expectancies (in all cases for P4T/in most cases for JTT) and increasing agreement with negative expectancies potentially associated with PO use over time. The exception was the potential to get in trouble with parents, and yet the baseline scores for this item were already only a half-point from the full agreement, highlighting a ceiling effect limiting significant change. A recent longitudinal study of 3396 ethnically diverse high school students (9th and 10th grades) found that increases in positive substance use expectancies among those who never used each respective substance predicted increased odds of onset during late adolescence (11th and 12th grades; alcohol: odds ratio [OR]ß 7.73, P<.001; tobacco: ORß 5.58, P<.001; marijuana: ORß 2.49, P=.001) [62]. Mounting evidence has suggested that outcome expectancies associated with substance use play a role in the initiation and progression of drug use [77]. Thus, change in expectancies is an important target in general substance use prevention and likely for PO misuse prevention as well.

Similar to the ceiling effect for the negative expectancy potential to get in trouble with parents, youth in both conditions perceived great risks at baseline; thus, limiting room for statistically significant change. However, both programs encouraged youth to increase their negative expectations around PO use. A systematic review of risk and protective factors associated with nonmedical use of prescription drugs (including opioids) among youth in the United States identified perceived risks as among the strongest and most consistent contributing factors at the individual level [78]. Both programs effectively impacted youth’s assessment of the difficulty and one’s ability to refuse offers to misuse POs as well as requests for personally prescribed POs. Scores across these variables were already close to optimal at baseline with little room for change. However, both programs produced statistically significant changes.

The second key finding is that P4T outperformed JTT on knowledge and negative expectancy changes as well as perceived ease of use, ease of understanding, and how much youth liked using technology to access the program. Knowledge-based, stand-alone interventions do not prevent later substance use. Programs that combine knowledge-based components with social competence and social influence approaches have greater support for their efficacy [79]. In addition, the JTT program focused on opioids among other drugs. However, it is significant that the program was effective in helping youth attain and sustain mastery of this important content.

The P4T youth also performed better than the JTT in increasing negative expectancy changes. Cross-sectional and longitudinal research findings indicate that the level of negative substance use expectancies is negatively associated with substance use behavior and use onset [80]. As noted above, although expectancy change and its effect on substance use onset may be more salient for positive expectancies (relative to negative), including both proximal and distal, negative expectancy items may increase the power of negative expectancies to impact substance use as powerfully as positive expectancies. The P4T program enlisted the stories of 8 youths detailing countless negative effects, both proximal and distal, of their experiences with NMUPO. It is possible that the level of detail with respect to negative effects through long periods of time (in most instances) served to more significantly impact the P4T scores in this area. The expectation-experience discrepancy (violating expectations) literature suggests that although it may seem reasonable in prevention efforts to stress the physical, psychological, and other harms of using drugs, this approach requires caution. Some marijuana-focused evidence suggests that if adolescents choose to use marijuana and the promised harms fall short of the severity they were led to expect; expectation changes may strengthen usage intentions [72]. However, legitimate opioid use before high school graduation is independently associated with a 33% increase in the risk of future opioid misuse after high school, and this association is concentrated among individuals who have little to no history of drug use and strong disapproval of illegal drug use at baseline [4]. It is highly plausible that expectation-experience discrepancies combined with the highly reinforcing properties of POs may have functioned to alter attitudes in this study.

Finally, the youth significantly preferred the P4T program to the JTT program in terms of ease of use cannot be underestimated. Drug prevention programs are arguably not at the top of the average adolescent’s list of preferred activities. Thus, lowering the barriers to a positive user experience in terms of program navigability and applicability to the lives of users is advantageous.

We drew a predominantly White, female, highly educated, middle adolescent (15-17 years), US sample, some of whom had personal histories of medical use of POs, friends who used POs nonmedically, and/or reported poor mental health. Aside from the highly educated [81,82] and White [83] characterizations of the sample, reporting past year medical use of POs, having friends who engage in NMUPO, and poor mental health are all often noted adolescent risk factors for NMUPO. The third key finding is that many youths in these latter 3 subgroups compared with youth not in the groups (comparators) scored consistently higher in terms of risk at baseline across variables of interest. Across all 3 subgroups (past year medical use of POs, having friends who engage in NMUPO, and poor mental health), intentions to use POs were significantly greater than those of comparators (P<.05). Youth with poor mental health and friends who engage in NMUPO underscored their elevated risk status by scoring almost all positive expectancies at a significantly higher rate than comparators, and perceived greater difficulty and lower ability to refuse offers and requests to use POs relative to comparators. These data support the growing evidence characterizing youth at greater risk for PO misuse. In addition, youth with past year medical use of POs and friends who engage in NMUPO tended toward disagreement with several negative expectancies relative to comparators (trouble with parents and do poorly in school). Those with a history of past year medical use of POs were in greater agreement than the comparators that POs might make a person feel sleepy. Thus, youth who had a history of past year medical use of POs may have more realistic expectations given their actual experience with the medications and be aware that POs do make you sleepy and that schoolwork does not have to suffer. Notably, youth who had past year medical use of Pos also scored only one positive expectancy (reduce pain) significantly higher relative to comparators. Consistent with the realistic expectations noted above, youth with medication experience may be aware that POs do indeed reduce pain. Although the evidence supports their at-risk status [4], the scores for youth who had past year medical use of POs function as a control on the other at-risk subgroups, as their differences from comparators are all consistent with having experienced the real consequences of past year medical use of POs.

The fourth key finding is that neither of the programs performed well in changing negative expectancy scores for overdose. Youth hovered around not sure across time points and programs for the negative expectancy of overdose, indicating that the risk of overdose does not seem to be a particularly salient concern in this sample. This is alarming and highlights an area where both programs need to do a better job of making this very real risk clearer. Between 1999 and 2016, the pediatric mortality rate from prescription and illicit opioids increased by 262.8% (N=approximately 9000) [84]. Annual rates were the highest for youth between 15 years and 19 years: 6755 (85.3%) of the deaths were unintentional, whereas 381 (4.8%) were attributed to suicide. In this group, the rates of fatal heroin, PO, and synthetic opioid poisonings increased by 404.8%, 94.7%, and 2925.0%, respectively. From 2014 to 2016, there were 1508 opioid deaths among adolescents between ages 15 and 19 years; of these, 468 (31.0%) were attributed to synthetic opioids (eg, fentanyl). One positive finding for one at-risk subgroup identified by our analyses was that youth with poor mental health scored significantly higher at baseline in agreement with the expectancy of overdose. Youth with poor mental health are at an increased risk of suicide [85]; thus, increasing awareness of the risks associated with NMUPO among this subgroup is reassuring. A 2018 survey of pediatric residents (N=69) examined knowledge, attitudes, and barriers to overdose prevention in the clinical setting. In total, 82% reported frequent exposure to pediatric patients using opioids and at risk of overdose. Although 94% of the trainees felt they had the responsibility to educate patients about overdose risk, only 42% discussed overdose prevention, and a mere 10% ever prescribed naloxone [86]. These findings reinforce that programs such as P4T and JTT must improve messaging around the potential for overdose when using POs medically and nonmedically.

Limitations and Future Research

This study has several limitations. Recruitment advertisements were directed toward parents (after advertisements directed toward youth failed to yield interested families) skewing the sample in favor of youth from homes with parents attentive to substance misuse issues, that is, youth potentially less likely to have problems with substances or at least youth with concerned parents, impacting the generalizability of the sample. Participants were also not blinded; thus, knowledge of group assignment may have affected their behavior in the trial (eg, motivation to access the control group website) as well as their responses to subjective outcome measures. In addition, the knowledge questions posed to participants in both groups were derived from content in the P4T program and might have consequently inflated the differences between groups at multiple time points. Finally, although the measures used in this study have been used in previous studies, one limitation is that they have not been clinically validated.

Improving prevention focused on the risk factors for opioid misuse and OUD is one of the 4 central areas proposed by the National Institute on Drug Abuse, which is necessary for reversing the opioid crisis [87]. The US opioid crisis is now in its third wave [88] of opioid overdose-related deaths (wave 1: POs, wave 2: heroin, wave 3: synthetic opioids, eg, illicit fentanyl). As the nature of opioids driving the crisis evolves, prevention efforts will also need to evolve to respond effectively. On the basis of the results of our study, future research might include an enhanced approach to target at-risk subgroups (eg, poor mental health, having friends who engage in NMUPO, past year medical use of POs), as the pattern of findings point to different risk factors based on the different risk profiles (we plan to publish the differential effects of group assignment on at-risk subgroups). Moreover, due to the ceiling effect we found with respect to perceived physical and other risks and the floor effect for intentions to use, it may be useful to recruit youth who may already be experimenting and/or who endorse increased sensation-seeking or self-medicating profiles [89]. Recognizing the heterogeneity among NMUPO will enhance the evolution and efficient tailoring of prevention programs. Future research may also require an enhanced approach to target the misuse of synthetic opioids such as illicit fentanyl by youth, as underscored by the dramatic increase in death rates noted above. Another potential direction for future research is to attend to how parents contribute to or can protect against adolescent NMUPO. Parental NMUPO is associated with adolescent NMUPO [90], and parental use of opioids is associated with a doubling of the risk of a suicide attempt by their children (OR 1.99, 95% CI 1.71-2.33) [91]. However, parental involvement and disapproval of substance use [92] and monitoring as well as reduced conflict [90] are associated with reduced odds of lifetime NMUPO.

Public Health Implications

The results suggest that the P4T program, which is freely available and accessible on the internet, may help prevent future misuse of PO among adolescents. Digital technology may provide scalable, effective prevention of PO misuse among teens.

Conflicts of Interest

LM is affiliated with both HealthSim, LLC (the business that received the NIH STTR grant and developed the web-based intervention described in this manuscript) as well as Dartmouth College. This relationship is extensively managed by her academic institution, Dartmouth College. JB is a member of the board of directors and treasurer of MySafeRx Inc., a non-profit scientific research organization, and does not receive any form of compensation (financial or otherwise) from this organization. All other authors have no conflicts to declare.

Multimedia Appendix 1

CONSORT-EHEALTH checklist (V 1.6.1).

PDF File (Adobe PDF File), 96 KB

Multimedia Appendix 2

Primary and secondary outcome (means).

DOCX File , 33 KB

Multimedia Appendix 3

Primary and secondary outcome (mixed models).

DOCX File , 44 KB


  1. McCabe SE, West BT, Veliz P, McCabe VV, Stoddard SA, Boyd CJ. Trends in Medical and Nonmedical Use of Prescription Opioids Among US Adolescents: 1976-2015. Pediatrics 2017 Apr;139(4) [FREE Full text] [CrossRef] [Medline]
  2. Kann L, McManus T, Harris WA, Shanklin SL, Flint KH, Queen B, et al. Youth Risk Behavior Surveillance - United States, 2017. MMWR Surveill Summ 2018 Dec 15;67(8):1-114 [FREE Full text] [CrossRef] [Medline]
  3. Murphy SM, Friesner DL, Rosenman R. Opioid misuse among adolescents: new evidence from a misclassification analysis. Appl Health Econ Health Policy 2015 Apr;13(2):181-192 [FREE Full text] [CrossRef] [Medline]
  4. Miech R, Johnston L, O'Malley PM, Keyes KM, Heard K. Prescription Opioids in Adolescence and Future Opioid Misuse. Pediatrics 2015 Nov;136(5):e1169-e1177 [FREE Full text] [CrossRef] [Medline]
  5. Austic E, McCabe SE, Stoddard SA, Ngo QE, Boyd C. Age and Cohort Patterns of Medical and Nonmedical Use of Controlled Medication Among Adolescents. J Addict Med 2015;9(5):376-382 [FREE Full text] [CrossRef] [Medline]
  6. McCabe SE, West BT, Morales M, Cranford JA, Boyd CJ. Does early onset of non-medical use of prescription drugs predict subsequent prescription drug abuse and dependence? Results from a national study. Addiction 2007 Dec;102(12):1920-1930 [FREE Full text] [CrossRef] [Medline]
  7. Dash GF, Wilson AC, Morasco BJ, Feldstein Ewing SW. A Model of the Intersection of Pain and Opioid Misuse in Children and Adolescents. Clin Psychol Sci 2018 Sep;6(5):629-646 [FREE Full text] [CrossRef] [Medline]
  8. Volkow ND. Teen prescription drug abuse a major health concern. Tenn Med 2009 Apr;102(4):28-29 [FREE Full text] [Medline]
  9. Substance Abuse and Mental Health Services Administration. Results from the 2013 national survey on drug use and health: summary of national findings. NSDUH Series H-48, HHS Publication No. (SMA). Rockville, MD: SAMHSA; 2014:14.
  10. Center for Behavioral Health Statistics and Quality. 2015 National Survey on Drug Use and Health: Detailed tables. 2016.   URL: [accessed 2021-01-01]
  11. Wu L, Pilowsky DJ, Patkar AA. Non-prescribed use of pain relievers among adolescents in the United States. Drug Alcohol Depend 2008 Apr 01;94(1-3):1-11 [FREE Full text] [CrossRef] [Medline]
  12. Boyd CJ, Esteban McCabe S, Teter CJ. Medical and nonmedical use of prescription pain medication by youth in a Detroit-area public school district. Drug Alcohol Depend 2006 Jan 04;81(1):37-45 [FREE Full text] [CrossRef] [Medline]
  13. McCabe SE, Boyd CJ, Young A. Medical and nonmedical use of prescription drugs among secondary school students. J Adolesc Health 2007 Jan;40(1):76-83 [FREE Full text] [CrossRef] [Medline]
  14. Sung H, Richter L, Vaughan R, Johnson PB, Thom B. Nonmedical use of prescription opioids among teenagers in the United States: trends and correlates. J Adolesc Health 2005 Jul;37(1):44-51. [CrossRef] [Medline]
  15. Whiteside LK, Walton MA, Bohnert ASB, Blow FC, Bonar EE, Ehrlich P, et al. Nonmedical prescription opioid and sedative use among adolescents in the emergency department. Pediatrics 2013 Nov;132(5):825-832 [FREE Full text] [CrossRef] [Medline]
  16. Muhuri PK, Gfroerer JC, Davies MC. Associations of Nonmedical Pain Reliever Use and Initiation of Heroin Use in the United States. CBHSQ Data Review.: Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality; 2013.   URL: [accessed 2021-01-01]
  17. Cicero TJ, Ellis MS, Surratt HL, Kurtz SP. The changing face of heroin use in the United States: a retrospective analysis of the past 50 years. JAMA Psychiatry 2014 Jul 01;71(7):821-826. [CrossRef] [Medline]
  18. Mars SG, Bourgois P, Karandinos G, Montero F, Ciccarone D. "Every 'never' I ever said came true": transitions from opioid pills to heroin injecting. Int J Drug Policy 2014 Mar;25(2):257-266 [FREE Full text] [CrossRef] [Medline]
  19. Palamar JJ, Shearston JA, Dawson EW, Mateu-Gelabert P, Ompad DC. Nonmedical opioid use and heroin use in a nationally representative sample of us high school seniors. Drug Alcohol Depend 2016 Jan 01;158:132-138 [FREE Full text] [CrossRef] [Medline]
  20. Curtin SC, Tejada-Vera B, Warmer M. Drug Overdose Deaths Among Adolescents Aged 15-19 in the United States: 1999-2015. NCHS Data Brief 2017 Aug(282):1-8 [FREE Full text] [Medline]
  21. Han B, Compton WM, Jones CM, Cai R. Nonmedical Prescription Opioid Use and Use Disorders Among Adults Aged 18 Through 64 Years in the United States, 2003-2013. JAMA 2015 Oct 13;314(14):1468-1478. [CrossRef] [Medline]
  22. Jones CM. The paradox of decreasing nonmedical opioid analgesic use and increasing abuse or dependence - An assessment of demographic and substance use trends, United States, 2003-2014. Addict Behav 2017 Feb;65:229-235. [CrossRef] [Medline]
  23. Hadland SE, Wharam JF, Schuster MA, Zhang F, Samet JH, Larochelle MR. Trends in Receipt of Buprenorphine and Naltrexone for Opioid Use Disorder Among Adolescents and Young Adults, 2001-2014. JAMA Pediatr 2017 Aug 01;171(8):747-755 [FREE Full text] [CrossRef] [Medline]
  24. Moore SK, Grabinski M, Bessen S, Borodovsky JT, Marsch LA. Web-Based Prescription Opioid Abuse Prevention for Adolescents: Program Development and Formative Evaluation. JMIR Form Res 2019 Jul 19;3(3):e12389 [FREE Full text] [CrossRef] [Medline]
  25. Marsch LA, Borodovsky JT. Technology-based Interventions for Preventing and Treating Substance Use Among Youth. Child Adolesc Psychiatr Clin N Am 2016 Dec;25(4):755-768. [CrossRef] [Medline]
  26. Marsch LA, Guarino H, Grabinski MJ, Syckes C, Dillingham ET, Xie H, et al. Comparative Effectiveness of Web-Based vs. Educator-Delivered HIV Prevention for Adolescent Substance Users: A Randomized, Controlled Trial. J Subst Abuse Treat 2015 Dec;59:30-37 [FREE Full text] [CrossRef] [Medline]
  27. Palamar JJ, Shearston JA, Dawson EW, Mateu-Gelabert P, Ompad DC. Nonmedical opioid use and heroin use in a nationally representative sample of us high school seniors. Drug Alcohol Depend 2016 Jan 01;158:132-138 [FREE Full text] [CrossRef] [Medline]
  28. Das JK, Salam RA, Arshad A, Finkelstein Y, Bhutta ZA. Interventions for Adolescent Substance Abuse: An Overview of Systematic Reviews. J Adolesc Health 2016 Oct;59(4S):S61-S75 [FREE Full text] [CrossRef] [Medline]
  29. Binder C. Behavioral fluency: Evolution of a new paradigm. Behav Anal 1996;19(2):163-197 [FREE Full text] [Medline]
  30. Schepis TS, Krishnan-Sarin S. Sources of prescriptions for misuse by adolescents: differences in sex, ethnicity, and severity of misuse in a population-based study. J Am Acad Child Adolesc Psychiatry 2009 Aug;48(8):828-836 [FREE Full text] [CrossRef] [Medline]
  31. Hudgins JD, Porter JJ, Monuteaux MC, Bourgeois FT. Prescription opioid use and misuse among adolescents and young adults in the United States: A national survey study. PLoS Med 2019 Nov;16(11):e1002922 [FREE Full text] [CrossRef] [Medline]
  32. McCabe SE, Cranford JA, Boyd CJ, Teter CJ. Motives, diversion and routes of administration associated with nonmedical use of prescription opioids. Addict Behav 2007 Mar;32(3):562-575 [FREE Full text] [CrossRef] [Medline]
  33. Boyd CJ, McCabe SE, Cranford JA, Young A. Prescription drug abuse and diversion among adolescents in a southeast Michigan school district. Arch Pediatr Adolesc Med 2007 Mar;161(3):276-281. [CrossRef] [Medline]
  34. Netemeyer R, Burton S, Delaney B, Hijjawi G. The Legal High: Factors Affecting Young Consumers' Risk Perceptions and Abuse of Prescription Drugs. Journal of Public Policy & Marketing 2015 Apr;34(1):103-118. [CrossRef]
  35. Daniulaityte R, Falck R, Carlson RG. "I'm not afraid of those ones just 'cause they've been prescribed": perceptions of risk among illicit users of pharmaceutical opioids. Int J Drug Policy 2012 Sep;23(5):374-384 [FREE Full text] [CrossRef] [Medline]
  36. Lord S, Marsch LA. Emerging trends and innovations in the identification and management of drug use among adolescents and young adults. Adolesc Med State Art Rev 2011 Dec;22(3):649-69, xiv [FREE Full text] [Medline]
  37. Marsch LA, Gustafson DH. The Role of Technology in Health Care Innovation: A Commentary. J Dual Diagn 2013;9(1):101-103 [FREE Full text] [CrossRef] [Medline]
  38. Ellickson P. Schools. In: Handbook on drug abuse prevention: a comprehensive strategy to prevent the abuse of alcohol and other drugs. Needham Heights, MA: Allyn & Bacon; 1995.
  39. Carey KB, Scott-Sheldon LAJ, Elliott JC, Bolles JR, Carey MP. Computer-delivered interventions to reduce college student drinking: a meta-analysis. Addiction 2009 Nov;104(11):1807-1819 [FREE Full text] [CrossRef] [Medline]
  40. Pradhan AM, Park L, Shaya FT, Finkelstein J. Consumer Health Information Technology in the Prevention of Substance Abuse: Scoping Review. J Med Internet Res 2019 Jan 30;21(1):e11297 [FREE Full text] [CrossRef] [Medline]
  41. Anderson M, Jiang J. Teens, Social Media & Technology 2018. Pew Research Center. 2018.   URL: 2018/05/31/teens-social-media-technology-2018/ [accessed 2021-01-01]
  42. Nesvåg S, McKay JR. Feasibility and Effects of Digital Interventions to Support People in Recovery From Substance Use Disorders: Systematic Review. J Med Internet Res 2018 Aug 23;20(8):e255 [FREE Full text] [CrossRef] [Medline]
  43. Carey KB, Scott-Sheldon LAJ, Elliott JC, Bolles JR, Carey MP. Computer-delivered interventions to reduce college student drinking: a meta-analysis. Addiction 2009 Nov;104(11):1807-1819 [FREE Full text] [CrossRef] [Medline]
  44. Champion KE, Newton NC, Barrett EL, Teesson M. A systematic review of school-based alcohol and other drug prevention programs facilitated by computers or the internet. Drug Alcohol Rev 2013 Mar;32(2):115-123. [CrossRef] [Medline]
  45. Walton MA, Resko S, Barry KL, Chermack ST, Zucker RA, Zimmerman MA, et al. A randomized controlled trial testing the efficacy of a brief cannabis universal prevention program among adolescents in primary care. Addiction 2014 May;109(5):786-797 [FREE Full text] [CrossRef] [Medline]
  46. Marsch LA, Bickel WK, Grabinski MJ. Application of interactive, computer technology to adolescent substance abuse prevention and treatment. Adolesc Med State Art Rev 2007 Aug;18(2):342-56, xii. [Medline]
  47. Schwinn TM, Schinke SP, Di Noia J. Preventing drug abuse among adolescent girls: outcome data from an internet-based intervention. Prev Sci 2010 Mar;11(1):24-32 [FREE Full text] [CrossRef] [Medline]
  48. Schinke SP, Fang L, Cole KC, Cohen-Cutler S. Preventing substance use among Black and Hispanic adolescent girls: results from a computer-delivered, mother-daughter intervention approach. Subst Use Misuse 2011;46(1):35-45 [FREE Full text] [CrossRef] [Medline]
  49. Newton NC, Teesson M, Vogl LE, Andrews G. Internet-based prevention for alcohol and cannabis use: final results of the Climate Schools course. Addiction 2010 Apr;105(4):749-759. [CrossRef] [Medline]
  50. Gulliver A, Farrer L, Chan JKY, Tait RJ, Bennett K, Calear AL, et al. Technology-based interventions for tobacco and other drug use in university and college students: a systematic review and meta-analysis. Addict Sci Clin Pract 2015 Feb 24;10:5 [FREE Full text] [CrossRef] [Medline]
  51. Marsch LA, Moore SK, Borodovsky JT, Solhkhah R, Badger GJ, Semino S, et al. A randomized controlled trial of buprenorphine taper duration among opioid-dependent adolescents and young adults. Addiction 2016 Aug;111(8):1406-1415 [FREE Full text] [CrossRef] [Medline]
  52. Dougherty KM, Johnston JM. Overlearning, fluency, and automaticity. Behav Anal 1996;19(2):289-292 [FREE Full text] [Medline]
  53. Johnson KR, Layng TV. Breaking the structuralist barrier. Literacy and numeracy with fluency. Am Psychol 1992 Nov;47(11):1475-1490. [Medline]
  54. Meier EA, Troost JP, Anthony JC. Extramedical use of prescription pain relievers by youth aged 12 to 21 years in the United States: national estimates by age and by year. Arch Pediatr Adolesc Med 2012 Sep 01;166(9):803-807 [FREE Full text] [CrossRef] [Medline]
  55. Cantrell MA, Lupinacci P. Methodological issues in online data collection. J Adv Nurs 2007 Dec;60(5):544-549. [CrossRef] [Medline]
  56. Ahern NR. Using the Internet to conduct research. Nurse Res 2005;13(2):55-70. [CrossRef] [Medline]
  57. Drug Enforcement Administration. DEA Launches Teen Web Page: Urges Teens to “Just Think Twice,” About Meth and Marijuana “It’s not All Right!”. Drug Enforcement Administration. 2005 Aug 30.   URL: [accessed 2021-01-01]
  58. Voepel-Lewis T, Boyd CJ, McCabe SE, Zikmund-Fisher BJ, Malviya S, Grant J, et al. Deliberative Prescription Opioid Misuse Among Adolescents andEmerging Adults: Opportunities for Targeted Interventions. J Adolesc Health 2018 Nov;63(5):594-600 [FREE Full text] [CrossRef] [Medline]
  59. Cooper ML, Kuntsche E, Levitt A, Barber L, Wolf S. Motivational models of substance use: A review of theory and research on motives for using alcohol, marijuana, and tobacco. In: The Oxford Handbook of Substance Use and Substance Use Disorders. New York, NY: The Oxford University Press; 2016:A.
  60. Jones BT, Corbin W, Fromme K. A review of expectancy theory and alcohol consumption. Addiction 2001 Jan;96(1):57-72. [CrossRef] [Medline]
  61. Callas PW, Flynn BS, Worden JK. Potentially modifiable psychosocial factors associated with alcohol use during early adolescence. Addict Behav 2004 Nov;29(8):1503-1515. [CrossRef] [Medline]
  62. Montes KS, Witkiewitz K, Pearson MR, Leventhal AM. Alcohol, tobacco, and marijuana expectancies as predictors of substance use initiation in adolescence: A longitudinal examination. Psychol Addict Behav 2019 Feb;33(1):26-34 [FREE Full text] [CrossRef] [Medline]
  63. Johnston L, Miech R, O'Malley P, Bachman J, Schulenberg J, Patrick M. (2020). Monitoring the Future national survey results on drug use 1975- 2019;Overview, key findings on adolescent drug use. Ann Arbor. [CrossRef]
  64. Bachman J, Johnston L, O'Malley P, Schulenberg JE. The Monitoring the Future project after thirty-seven years: Design and procedures. In: (Monitoring the Future Occasional Paper No. 76). Ann Arbor, MI: Institute for Social Research; 2011:93.
  65. Arria AM, Caldeira KM, Vincent KB, O'Grady KE, Wish ED. Perceived harmfulness predicts nonmedical use of prescription drugs among college students: interactions with sensation-seeking. Prev Sci 2008 Sep;9(3):191-201 [FREE Full text] [CrossRef] [Medline]
  66. Terry-McElrath YM, O'Malley PM, Patrick ME, Miech RA. Risk is still relevant: Time-varying associations between perceived risk and marijuana use among US 12th grade students from 1991 to 2016. Addict Behav 2017 Nov;74:13-19 [FREE Full text] [CrossRef] [Medline]
  67. Yeomans-Maldonado G, Patrick ME. The Effect of Perceived Risk on the Combined Used of Alcohol and Marijuana: Results from Daily Surveys. Addict Behav Rep 2015 Dec 01;2:33-36 [FREE Full text] [CrossRef] [Medline]
  68. Parker MA, Anthony JC. Population-level predictions from cannabis risk perceptions to active cannabis use prevalence in the United States, 1991–2014. Addictive Behaviors 2018 Jul;82:101-104 [FREE Full text] [CrossRef]
  69. Schwinn TM, Schinke SP, Di Noia J. Preventing drug abuse among adolescent girls: outcome data from an internet-based intervention. Prev Sci 2010 Mar;11(1):24-32 [FREE Full text] [CrossRef] [Medline]
  70. Fearnow-Kenney M, Hansen WB, McNeal RB. Comparison of Psychosocial Influences on Substance Use in Adolescents: Implications for Prevention Programming. Journal of Child & Adolescent Substance Abuse 2002 Jun;11(4):1-24. [CrossRef]
  71. Haerens L, Deforche B, Vandelanotte C, Maes L, De Bourdeaudhuij I. Acceptability, feasibility and effectiveness of a computer-tailored physical activity intervention in adolescents. Patient Educ Couns 2007 Jun;66(3):303-310 [FREE Full text] [CrossRef] [Medline]
  72. Skenderian JJ, Siegel JT, Crano WD, Alvaro EE, Lac A. Expectancy change and adolescents' intentions to use marijuana. Psychol Addict Behav 2008 Dec;22(4):563-569 [FREE Full text] [CrossRef] [Medline]
  73. Armitagexclusions, losses TFICT. Shapiro L. T. editor. Clinical Trials, Issues and Approaches. New York: Marcel Dekker, Inc; 1983:E.
  74. Marsch LA, Bickel WK, Badger GJ. Applying Computer Technology to Substance Abuse Prevention Science: Results of a Preliminary Examination. Journal of Child & Adolescent Substance Abuse 2007 Mar 06;16(2):69-94. [CrossRef]
  75. Julious S. Sample Sizes for Clinical Trials. Chapman & Hall/CRC. Boca Raton, FL 2010. [CrossRef]
  76. Chow S, Shao J, Wang H. 2008. Sample Size Calculations in Clinical Research (Second Edition). Chapman & Hall/CRC. Boca Raton, FL 2008.
  77. Jones BT, Corbin W, Fromme K. A review of expectancy theory and alcohol consumption. Addiction 2001 Jan;96(1):57-72. [CrossRef] [Medline]
  78. Nargiso JE, Ballard EL, Skeer MR. A systematic review of risk and protective factors associated with nonmedical use of prescription drugs among youth in the United States: a social ecological perspective. J Stud Alcohol Drugs 2015 Jan;76(1):5-20. [Medline]
  79. Faggiano F, Minozzi S, Versino E, Buscemi D. Universal school-based prevention for illicit drug use. Cochrane Database Syst Rev 2014(12):CD003020. [CrossRef] [Medline]
  80. Doran N, Khoddam R, Sanders PE, Schweizer CA, Trim RS, Myers MG. A prospective study of the Acquired Preparedness Model: the effects of impulsivity and expectancies on smoking initiation in college students. Psychol Addict Behav 2013 Sep;27(3):714-722 [FREE Full text] [CrossRef] [Medline]
  81. Agrawal A, Gardner CO, Prescott CA, Kendler KS. The differential impact of risk factors on illicit drug involvement in females. Soc Psychiatry Psychiatr Epidemiol 2005 Jun;40(6):454-466. [CrossRef] [Medline]
  82. Martins SS, Kim JH, Chen L, Levin D, Keyes KM, Cerdá M, et al. Nonmedical prescription drug use among US young adults by educational attainment. Soc Psychiatry Psychiatr Epidemiol 2015 May;50(5):713-724 [FREE Full text] [CrossRef] [Medline]
  83. Ford JA, Rigg KK. Racial/Ethnic differences in factors that place adolescents at risk for prescription opioid misuse. Prev Sci 2015 Jul;16(5):633-641. [CrossRef] [Medline]
  84. Gaither JR, Shabanova V, Leventhal JM. US National Trends in Pediatric Deaths From Prescription and Illicit Opioids, 1999-2016. JAMA Netw Open 2018 Dec 07;1(8):e186558 [FREE Full text] [CrossRef] [Medline]
  85. Gomes AP, Soares ALG, Kieling C, Rohde LA, Gonçalves H. Mental disorders and suicide risk in emerging adulthood: the 1993 Pelotas birth cohort. Rev Saude Publica 2019;53:96 [FREE Full text] [CrossRef] [Medline]
  86. Wilson JD, Berk J, Adger H, Feldman L. Identifying Missed Clinical Opportunities in Delivery of Overdose Prevention and Naloxone Prescription to Adolescents Using Opioids. J Adolesc Health 2018 Aug;63(2):245-248 [FREE Full text] [CrossRef] [Medline]
  87. Volkow ND, Jones EB, Einstein EB, Wargo EM. Prevention and Treatment of Opioid Misuse and Addiction. JAMA Psychiatry 2019 Feb 01;76(2):208. [CrossRef]
  88. Ciccarone D. The triple wave epidemic: Supply and demand drivers of the US opioid overdose crisis. Int J Drug Policy 2019 Sep;71:183-188 [FREE Full text] [CrossRef] [Medline]
  89. Young A, McCabe SE, Cranford JA, Ross-Durow P, Boyd CJ. Nonmedical use of prescription opioids among adolescents: subtypes based on motivation for use. J Addict Dis 2012;31(4):332-341 [FREE Full text] [CrossRef] [Medline]
  90. Griesler PC, Hu M, Wall MM, Kandel DB. Nonmedical Prescription Opioid Use by Parents and Adolescents in the US. Pediatrics 2019 Mar;143(3). [CrossRef] [Medline]
  91. Brent DA, Hur K, Gibbons RD. Association Between Parental Medical Claims for Opioid Prescriptions and Risk of Suicide Attempt by Their Children. JAMA Psychiatry 2019 May 22. [CrossRef] [Medline]
  92. Monnat SM, Rigg KK. Examining Rural/Urban Differences in Prescription Opioid Misuse Among US Adolescents. J Rural Health 2016;32(2):204-218 [FREE Full text] [CrossRef] [Medline]

CHIT: consumer health information technology
DEA: Drug Enforcement Administration
JTT: JustThinkTwice
NMUPO: nonmedical use of prescription opioids
OR: odds ratio
OUD: opioid use disorder
PO: prescription opioid
P4T: POP4Teens

Edited by G Eysenbach; submitted 28.02.20; peer-reviewed by T Schwinn, D Leightley; comments to author 26.10.20; revised version received 21.12.20; accepted 13.01.21; published 25.02.21


©Lisa A Marsch, Sarah K Moore, Michael Grabinski, Sarah Y Bessen, Jacob Borodovsky, Emily Scherer. Originally published in JMIR Public Health and Surveillance (, 25.02.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.