Efficacy of Web-Based Collection of Strength-Based Testimonials for Text Message Extension of Youth Suicide Prevention Program: Randomized Controlled Experiment

Introduction

Background and Motivation

Public health practitioners and researchers are increasingly harnessing mobile technology to enhance the reach and effectiveness of evidence-based interventions [1-3]. Equipping members of a target population to deliver effective public health messaging to peers is an established approach in substance abuse prevention, human immunodeficiency virus prevention, and health promotion [4-7], and is promising for suicide prevention [8]. The effectiveness of peer-to-peer public health messaging strategies is congruent with peer-to-peer influences on behavior [10]. For youth with acute distress and with numerous risk factors for suicide, social modeling of emotional regulation strategies could slow or reverse a negative trajectory or prevent impulsive attempts to escape painful emotions [11,12]. A crucial barrier to technological extensions of these so-called ‘opinion leader’ interventions is the challenge of helping peer leaders to deliver effective messages to peers remotely. This requires development of Web-based replacements for typical in-person components [9]. In this study, we tested a Web-based interface for remotely helping high school peer leaders to produce high-quality, personal testimonials for use in a text messaging extension of an evidence-based, peer-led suicide prevention program.

Sources of Strength is a school-based suicide prevention program that is certified on the United States National Registry of Evidence-Based Programs and Practices. The program seeks to reduce risk and strengthen protective factors in the population by preparing diverse ‘key opinion leaders’ to conduct public health messaging and activities with peers to increase school-wide positive coping norms and communication with trusted adults, and bring suicidal peers to seek for adult help [8,13]. Currently implemented in over 3000 high schools and colleges across the United States, Sources of Strength was the first suicide prevention program to apply a peer ‘opinion leader’ approach. As part of this strategy, peer leaders publicly share personal stories (or testimonials) of “hope, help, and strength” with peers and encourage them to do the same. Those who share their personal stories act as distributed change agents, generating much of the intervention content received by their school’s population. In a randomized controlled trial conducted in 18 schools, trained peer leaders were 4 times more likely to refer a suicidal friend to an adult, and school-wide help-seeking norms improved in schools implementing Sources of Strength [8]. An ongoing trial with 40 schools in rural and underserved communities is testing the impact and proposed network diffusion model of Sources of Strength on reducing suicide attempts.

Text4Strength is a text messaging extension of Sources of Strength. It is currently under development with funding from the National Institute of Mental Health (K23MH101449). Text4Strength uses text-based peer testimonials and other types of text messages to introduce high school students to Sources of Strength concepts and extends Sources of Strength by also introducing emotional skills associated with decreased depressive symptoms and lower suicide risk [14-17]. Text messaging is a logical medium for disseminating messages because it is the dominant mode of communication for youth [18] and has strong empirical support for delivering health interventions [2,3,19-25]. The current version of Text4Strength specifically targets ninth graders because research indicates an increase in emotional and behavioral problems between ages 14 and 15, when most students are starting high school [26]. In addition to normal developmental changes resulting in heightened emotional activation [27-29] and diminished inhibitory control abilities [30], ninth graders face new challenges associated with the transition to high school, including greater responsibility for seeking help, greater emphasis on romantic relationships, and greater academic pressure.

In developing Text4Strength, we faced challenges that previous work in translating behavioral interventions into text (e.g., [9]) had not yet surmounted: collecting interesting, personal, and credible [31,32] testimonials consistent with the intervention model, goals, and targets. We addressed these challenges by developing Stories of Personal Resilience in Managing Emotions (StoryPRIME), a Web-based interface to guide peer leaders through the process of remembering, sharing, and condensing relevant personal stories into the pithy testimonials needed for text messaging. We developed StoryPRIME through an interdisciplinary design process in which suicide prevention researchers consulted with human-computer interaction (HCI) researchers. Together, we conducted design workgroups with high school peer leaders, as well as feasibility testing with an adult sample to ensure the safety of the StoryPRIME interface prior to testing it in a randomized controlled experiment with a teenage population. In this paper, we report on results from the randomized controlled experiment of StoryPRIME to discover whether or not Web-based eliciting could assist peer leaders in generating high-quality text message testimonials that are interesting, personal, and credible to ninth graders.

Development and Feasibility Testing

Suicide prevention and HCI researchers worked together to translate practices used in Sources of Strength to elicit peer leaders to tell their stories using a Web-based interface. In the Sources of Strength training program, student peer leaders participate in a 4-hour training in which helpful testimonials are modeled and practiced. After initial training, students participate in approximately biweekly meetings where adult advisors give them opportunities to practice and receive feedback sharing their “hope, help, and strength” stories with peers and encourage them to do the same [13]. The challenge we faced was translating this highly organic discussion and coaching process in Sources of Strength into a structured, Web-based interface.

To tackle this challenge, HCI researchers piloted an initial interface with our interdis­ciplinary team and other lab members, and then conducted a participatory design [33] session and workgroup with 7 high school peer leaders (4 females, 3 males). We met with the students 7 times, for 60-90 minutes each, to get their feedback on the design of the StoryPRIME interface for testimonial writing and in general, about our plans for the Text4Strength text messaging program. Using this feedback, HCI researchers developed several variations/prototypes of the StoryPRIME interface. We then asked the workgroup participants to write stories about themselves using these interfaces, and discuss the writing experience with the workgroup participants and our research group. Based on the workgroup’s feedback, we refined the StoryPRIME interface, partitioning the writing process into 3 main steps to help writers focus on the essential parts of the story: (1) challenge faced, (2) how they solved it, and (3) the outcome.

We pilot tested the StoryPRIME interface with an adult convenience sample in order to ensure feasibility and safety before testing with adolescent high school students. Fifty adults (29 females, 21 males; mean age of 30.28 (standard deviation=7.73)) were recruited from Mechanical Turk (MTurk)–a crowdsourcing website where people are paid to do Web-based tasks. Participants wrote testimonials about com­mon stressful situations and emotional skills and resources they used to cope. A separate sample of 61 adults recruited from MTurk rated the testimonials on how interesting, personal, genuine/credible, and relatable they were. In addition, 2 public health experts rated the stories on how appropriate and (potentially) effective they were. The results were promising: 70% (35/50) of the testimonials were judged as interesting, 78% (39/50) as personal, 80% (40/50) as credible, and 76% (38/50) as relatable. The two suicide prevention experts judged 76% (38/50) and 72% (36/50) of the testimonials as appropriate/safe and effective, respectively. Furthermore, qualitative responses from Web-based participants indicated that they found the experience rewarding and enjoyable, with no negative report of burden or emotional discomfort.

Methods

Participants

The participants included, testimonial writers: 36 students (23 females, 13 males) of grades 10-12 (ages 16-18) from 2 rural high schools in Western New York wrote testimonials. All trained Sources of Strength peer leaders in each school (approximately 45) were invited to participate and given parent information letters and permission forms to return; and raters: 24 (10 females, 14 males) ninth grade (ages 14-15) students whose parents gave them permission to participate in this study rated on the testimonials. Two classes of ninth grade students in each school (approximately 50) were invited via to participate via a school-wide announcement and given parent information letters and permission forms to return. All participants were paid US$10 as a thank you for their time. The University of Rochester institutional review board approved this study.

Measures

Students rated testimonials on the following dimensions using a 1-5 (‘Strongly Disagree’ to ‘Strongly Agree’) Likert scale: relatable (“My friends and I handle similar challenges”), useful/relevant (“This student’s solution/advice would work for me”), intriguing (“I am curious about more detail/background to this SPECIFIC experience”), and likable (“I would be interested in hearing more student experiences like this”).

Design

We tested the StoryPRIME interface using a double-blind, randomized controlled design. Writers were randomly assigned to write a personal story with the aid of StoryPRIME (N=18 with 12 females) or to a control condition (N=18 with 13 females). Within each school, we assigned an equal number of students into either StoryPRIME or control condition–7 per condition for one, and 11 per condition for another school. There was no opportunity for “contamination” because both groups worked on testimonials at the same time in a quiet room on separate computers–participants simply received different on-screen instructions and interface depending upon their condition. We asked student writers to write as many testimonial as they could during a 30-minute session. Most students (N=34) completed writing 2 testimonials while 2 of them wrote just one each in the allotted time. Testimonial writers collectively produced 70 testimonials, 35 in the StoryPRIME condition and 35 in the control condition. At the end of the writing session, we asked students to complete a brief Web-based anonymous survey for their feedback on their writing experience and StoryPRIME interface.

Testimonials were presented in identical format and random order to the ninth grade student raters (N=24 with 10 females) who were blind to the writer’s condition. Each testimonial was rated by at least 3 different students. Testimonial raters rated a random selection of 12 testimonials each, 6 in each condition. Ratings were conducted anonymously on private laptops of the raters and they received payment immediately after submitting their responses. In addition to explaining to the raters that we would not know who provided which ratings, instructions emphasized that we desired and needed their honest opinions–positive and negative.

Conditions

We met with student writers in their school’s computer lab where each student was assigned a computer. All student testimonial writers accessed a website developed for this study on which they were invited to write strength-based testimonials that would help and appeal to ninth grade students:

The Center for the Study and Prevention of Suicide works with high schools to teach teenagers emotional skills so that they stay strong when they face hard times. To do that, we need real examples from people who have used different strategies for healthy re­sponses to conflict or common challenges they face in high school.

Students then selected a challenge to write about from a list of common challenges faced by high school students, generated with the help of the student workgroup in the participatory design process (see Textbox 1).

StoryPRIME condition

Participants in the StoryPRIME condition were asked to:

Think of a time in high school when you handled or learned something about [the challenge user selected previously].

After proceeding, the participants saw:

Thank you! Now we will ask you THREE questions to help you write the details about the challenge (stress) you just selected. Please be as specific as possible. At the end, we will ask you to summarize your answers into a ~300-character story/testimonial.

If the participants chose the “Ready to Help” option, they were presented with a series of 3 writing prompts and accompanying tips. Figure 1 shows these writing prompts for the challenge of “Figuring out one’s ‘place’ or fitting in.” After completing all 3 steps, participants were shown all of their previous responses on a single page and asked to generate a concise summary (∼300 characters) as shown in Figure 2. Following completion, participants were asked to repeat the procedure for another challenge.

‎View this figure

‎View this figure

Control condition

Participants in the control condition were given simple instructions: “In the space below, please share a brief story (~300 characters) about [the challenge user se­lected previously], which might help another high school student. It’s OK to use abbreviations as you would in a txt.”

Statistical Analyses

Linear mixed-effects models were used to model the average of the 4 questions as a function of intervention (StoryPRIME vs control), controlling for rater via random effects. Similar models were used to model the Likert score for each individual question. Restricted maximum likelihood estimation via lme() in R ×64-bit version 3.1.1 was used to fit all models. We decided a priori in consultation with our biostatistician (DP) to use significance level of .10 in order to match the scientific needs of testing a novel procedure–that is, to mitigate the risk of throwing out a useful procedure based on an overly rigorous (and arbitrary) P level [34,35].

Results

Principal Findings

Ninth grade students (N=24), who were blind to the conditions, found testimonials written with the aid of StoryPRIME (N=35) more favorable than those written with the control interface (N=35). The mean of the 4 ratings–each using the 1-5 Likert scale where 5 is the maximum favorable score achievable–for testimonials written in the control condition was 3.47 (standard error (SE)=.10), and the mean for testimonials written in the StoryPRIME condition was 3.60 (SE=.10). The difference in mean ratings across conditions was .13 (SE=.08), P=.054. When questions were examined individually, student ratings on all 4 questions were in the same direction favoring the StoryPRIME condition over the control. As presented in Table 1, ninth grade students, who were blind to the conditions, found testimonials that had been written with the aid of StoryPRIME more relatable, useful/relevant, and likable than those written with the control interface.

Student Testimonials

The challenges students wrote about centered mostly on academic pressures (grades, tests), time management, and friends. Some were about fitting in and boy/girlfriend drama. Specifically, the breakdown of peer leader testimonials was as follows: 25 testimonials related to school pressure (12 control, 13 intervention); 19 about balancing time (11 control, 8 intervention); 8 regarding different class schedules from friends (4 control, 4 intervention); 3 about feeling confused (3 control); 4 about relationship drama (2 control, 2 intervention); 6 about fitting in (2 control, 4 intervention); 1 about getting cut in sports (1 control); 3 about not knowing how to get help from teachers and adults (3 intervention); and 1 about family/parent issues (1 intervention). Some of the testimonials written in StoryPRIME can be seen in Textbox 2.

The StoryPRIME interface prompts writers to describe how they applied some common coping strategies–such as family support, mentors, and positive friends as seen in the second prompt of Figure 1. Students wrote stories that mentioned relying on family support (1 in control, 2 in intervention); positive friends (5 control, 11 intervention); mentors (4 control, 14 intervention); sports (2 control, 0 intervention); and none (23 control, 8 intervention).

Most students in both conditions were happy to have shared their stories. For example, when asked:

How did you feel? Did you find anything upsetting or uncomfortable while writing the story?

One student in the control condition wrote:

It was an aha moment for me and realization.

Another student in the StoryPRIME condition wrote:

I'm happy that I was able to share my story, in order to provide advice for students who are entering high school and preparing for challenges, as well.

Discussion

Challenges, Achievements, and Feedback

The randomized controlled experiment produced promising results: (1) high school students assisted by StoryPRIME wrote numerous testimonials about diverse challenges such as school pressure (grades, tests), time management, friends, fitting in, and boy/girlfriend drama, (2) compared with the control condition, students assisted by StoryPRIME wrote testimonials that ninth graders found more relatable, useful/relevant, and likable, and (3) students had a positive experience writing testimonials using the StoryPRIME interface based on the overwhelmingly positive feedback they provided in the exit survey. Although the differences between the StoryPRIME testimonials and the control testimonials are small, they are meaningful–especially given the scale at which we could use StoryPRIME to reach potential student testimonial contributors.

Regarding the nonsignificant results with respect to whether messages were intriguing (“I am curious about more detail/background to this SPECIFIC experience”), we suspect that the suggested 300-character limit might not be sufficient enough to write intriguing (interesting) stories. Allowing students to write more characters may encourage them to include more details about how the event transpired, thereby making the story more intriguing. This goes in line with some suggestions from writers of both StoryPRIME and control condition such as:

Ask for more details

Add more pictures and fun questions such as how are you doing? do you feel okay today? etc.

...maybe just letting kids describe their situation more and how family was involved.

The downside of allowing more details is that the resulting stories may no longer fit the criteria of a short text message, which is what we originally set out to achieve for our text message campaign.

Feedback on StoryPRIME interface by student writers were especially encouraging; one of the writers in StoryPRIME condition wrote:

I thought this website (StoryPRIME) was great. I liked the fact that it helped me remember some times I struggled by giving me examples of things other students struggled with.

On the other hand, a writer in control condition suggested:

more room to write about something or more prompting questions for personalization.

Also, another wrote:

It's a bit hard to recall what happened over a years ago and decide which one to write about.

These comments and better ratings achieved by testimonials written using StoryPRIME suggest that the interface is helping students recall relevant topics in high school and write effective stories about them.

Despite the aforementioned positive feedback, the StoryPRIME interface is far from perfect. A few writers in StoryPRIME condition suggested potential improvements to the interface. One wrote:

You can improve this website by having us go into more depth of how we handled the situation.

Referring to the final summary step in StoryPRIME where the interface suggested students to limit their writing to 300 characters, another suggested:

(Allow) More characters to type your summary with.

Nevertheless, some of them provided positive feedback about the interface:

This website is awesome as it is :)

Honestly, I don't feel that this website needs any improvement. It is very easy to follow and helped me tell my stories.

In addition to helping students to write testimonials, StoryPRIME interface seemed to provide them with an opportunity to reflect and learn from their past experiences. One writer in StoryPRIME condition wrote:

it was easy and felt nice to describe an incident that helped me become a good student.

Another wrote:

...it was an eye opening experience. I learned that I can reflect back on my early high school career in detail.

Finally, one student wrote:

I learned that i actually learned from my mistakes as a freshmen to be more successful.

The StoryPRIME interface helped prevention researchers translate a key component of an existing in-person, evidence-based practice from a public health intervention (ie, eliciting Sources of Strength peer leaders to share their stories) onto a Web-based platform (ie, text messaging), which is a hurdle many evidence-based public health programs face as they seek to increase their reach through technology [2,36,37]. The ability to reliably generate short-form testimonials via StoryPRIME opens up new possibilities for other public health practitioners who wish to draw on peer models to deliver messages through short-form Web-based media, such as text messages and social media microblogs [38]. Ultimately, we plan to use StoryPRIME to elicit strength-based testimonials from members of the target population who are not peer leaders. By translating a school-based technique into a Web- and text messaging–based one, we hope to increase the reach of Text4Strength to an audience broader than the one reached by its parent program, Source of Strength, alone. However, it is worth nothing that in this experiment, we are testing the efficacy of the procedure in generating appealing and relevant testimonials, not the efficacy of the procedure in suicide prevention. The latter will be accomplished in a trial Text4Strength program (currently underway) to which StoryPRIME contributes.

Limitations

Several limitations should be noted. First, this trial was conducted in 2 small, rural schools in the United States. While we have no reason to expect students from different backgrounds to respond differently from these students, the interface still needs to be tested in a variety of environments and prevention programs. Moreover, the raters in both control and intervention groups might not be representative of all ninth graders in those schools. Second, while StoryPRIME produced more relatable, useful, and likable testimonials, the effect size of the difference between the StoryPRIME and the simple control conditions was not large, so practitioners need to weigh the benefit of using StoryPRIME with the cost of a longer procedure. Finally, a nontrivial portion of the testimonials participants wrote were not interesting or relevant, and could not be used in our program. Thus, we have a procedure that generates high-quality testimonials to choose from, but have not yet arrived at the ideal state in which our interface could elicit youth testimonials ready to be automatically shared without monitoring.

Comparison With Prior Work

While our work is the first to successfully translate an element of a school-based suicide prevention program onto a Web-based and text messaging–based platform, it builds on previous research on public health related to: (1) text messaging in public health interventions, (2) peer opinion leaders’ effectiveness in public health interventions, and (3) other public health efforts to use short form messaging. Previous work with text messaging in nonclinical settings involved smoking cessation [37], child and maternal health, and sexual health [23-25], but ours is the first effort to use texting in population-oriented suicide prevention, as we ultimately aim to do in Text4Strength. Using testimonials created with StoryPRIME in Text4Strength text messages is consistent with “Texting for Public Health Toolkit’s” strategy of crafting text messages with “engaging writing devices” [39]. While researchers have shown that peer leaders can effectively influence social norms and behavior in public health contexts, such as suicide prevention interventions [8], and in health care settings [6,7], our work is the first to translate the influence of peer leaders onto Web-based media in order to increase the reach of our intervention. This work is quite different from previous work that translated the content of a curriculum or treatment manual into text [37]. Rather, our work involved translating a complex training and coaching process that occurs in a school-based program into a Web-based interface, ultimately yielding text messages to convey public health messages. Finally, our work contributes to the growing body of research involving translating evidence-based practices into scalable Web-based formats in order to have a broader public health impact. Programs that involve community members could use a procedure like StoryPRIME to develop compelling testimonials to add to existing public health text messaging programs, such as those used increase adherence to malaria treatment [40,41].

Conclusions

StoryPRIME serves as an example for translating complex procedures from an in-person evidenced-based program and moving them to a Web-based one in order to scale up technology use. We translated the procedure for collecting peer-to-peer testimonials, a critical component of Sources of Strength. StoryPRIME elicited high school peer leaders to generate engaging testimonials suitable for use in text messaging. The next step is to incorporate StoryPRIME testimonials into Text4Strength, a text messaging extension of Sources of Strength, to determine whether the testimonials are effective in promoting positive coping and emotional skills. Other programs that disseminate peer testimonials in public health messaging can draw on this work to improve the efficiency and effectiveness of collecting brief testimonials for use in short-form, Web-based media, such as text messages and social media microblogs.