The Journal of Extension - www.joe.org

August 2017 // Volume 55 // Number 4 // Research In Brief // 4RIB9

Youths Perceive Some Improvement in Substance Abuse Prevention Knowledge, Skills, and Assets from Participation in 4-H Health Rocks!

Abstract
The 4-H Health Rocks! curriculum aims to reduce use of tobacco, alcohol, and other drugs and promote healthful lifestyle choices among 8- to 14-year-old youths. A retrospective "post-then-pre" survey of Tennessee participants was aimed at describing the demographic characteristics of participants and investigating respondents' perceptions of program outcomes. Although positive, significant results in youths' perceived knowledge, skills, and assets were found, the majority of youths reported no change from before program participation to after program participation. Recommendations include addressing the need for additional research that aligns respondents' perceptions with program delivery settings and the need to explore different evaluation approaches.


Kane T. Reeves
Extension Agent II
Cooperative Extension Program
Tennessee State University
Jackson, Tennessee
kreeves@tnstate.edu

Joseph L. Donaldson
Assistant Professor
Department of 4-H Youth Development, Agricultural Leadership, Education and Communications
The University of Tennessee
Knoxville, Tennessee
jldonaldson@tennessee.edu

Mitchell Naylor
Student Staff Assistant
Department of 4-H Youth Development, Agricultural Leadership, Education and Communications
University of Tennessee Martin
Martin, Tennessee
mnaylor5@tennessee.edu

Lisa LeBleu
Associate Professor
Family and Consumer Sciences
University of Tennessee Martin
Martin, Tennessee
llebleu@utm.edu

Introduction

The use of tobacco, alcohol, and other drugs among young people remains a national dilemma, with an estimated 23.9 million Americans 12 years old and older having used an illicit drug or abused a prescription medication (National Institute on Drug Abuse, 2014). Youth access to tobacco, alcohol, and other drugs is a considerable problem. In a 2012 study on the availability of marijuana among youths, 37% of eighth graders, 69% of 10th graders, and 82% of 12th graders reported that it was very easy to obtain marijuana (Johnston, O'Malley, Bachman, & Schulenberg, 2013). Almost all youths aged 12 to 14 are at risk for trying a substance due to peer pressure or the normalizing of substance use in their communities (Miller & Hendrie, 2008). Using tobacco, alcohol, and other drugs before the brain is fully developed increases the risk of addiction to other substances (National Council on Alcoholism and Drug Dependence Inc., 2015).

4-H Health Rocks! Program Delivery

Through the combined efforts of the National 4-H Council, its donors, and faculty of the land-grant university system, Cooperative Extension provides 4-H Health Rocks!, a program to prevent tobacco, alcohol, drug abuse among 8- to14-year-old youths.The program is delivered through 10 hr of instruction on the negative health outcomes of substance abuse, communication, decision making, relationship building, and self-esteem. The program includes a two-part curriculum, with a beginning level for youths aged 8 to 12 and an intermediate level for youths aged 12 to 14 (Donaldson & Crowe, 2013). The program is delivered in multiple settings, including residential camps, day camps, afterschool programs, and in-school club meetings. The program can be organized using various time frames; for example, an afterschool program may devote 1 hr to 4-H Health Rocks! each week for 10 weeks (Kumaran, Fogarty, Fung, & Terminello, 2015; National 4-H Council, 2009).

4-H Health Rocks! Program Evaluation

Kumaran et al. (2015) evaluated 4-H Health Rocks! through a qualitative study of Florida Extension 4-H professionals conducting the program. They identified time constraints, unsuccessful volunteer recruitment, and an inability to fulfill the program's 10 hr of implementation as major barriers to program success. An evaluation of the 4-H Health Rocks! program in Georgia indicated that the program produced positive, significant gains in youth participants' perceptions of their knowledge, beliefs/attitudes, skills, and behavioral intentions (Self, Morgan, Fuhrman, & Navarro, 2013).

The Georgia study (Self et al., 2013), as well as the research reported here, involved use of a retrospective "post-then-pre" questionnaire. The retrospective post-then-pre questionnaire is completed at the end of the program and asks a participant to evaluate his or her status in a particular realm (i.e., behavior) after participating in the program and then, in a second question, to evaluate what his or her status had been in that realm before participating in the program. Retrospective post-then-pre design helps respondents have "sufficient knowledge to answer the question validly" (Rockwell & Kohn, 1989, "Correcting Problem," para. 1), requires minimal time investment from both respondents in answering questions and researchers in analyzing data (Davis, 2003), and provides a more complete data set than traditional pretest/posttest approaches do (Raidl et al., 2004). However, the design has some important drawbacks. Retrospective post-then-pre design relies on self-reports, and respondents may provide the answers they think researchers want; respondents may not understand the concept because of cultural differences, language barriers, and/or literacy levels; and research is lacking regarding how to best administer post-then-pre questionnaires (Klatt & Taylor-Powell, 2005).

On the whole, the research reported here differed from the previous 4-H Health Rocks! evaluations. Unlike with the Florida study (Kumaran et al., 2015), a quantitative approach was used. And although a quantitative, retrospective post-then-pre design was used for both the Georgia study (Self et al., 2013) and the study reported here, different instruments were used in the two studies. The Georgia study involved a questionnaire with 36 items (Self et al., 2013) requiring two-part responses, whereas the research reported here involved a questionnaire with 13 items requiring two-part responses.

Purpose

The purpose of the study reported here was to describe results of a 4-H Health Rocks! survey of Tennessee program participants during the 2011–2012 program year. Specific objectives were

  1. to describe the demographic characteristics of respondents (gender, race, ethnicity, and grade) and
  2. to describe differences, if any, in the postprogram and retrospectively identified preprogram scores of respondents regarding their knowledge, skills, and assets.

Methods

During the 2010–2011 academic year, a retrospective post-then-pre survey was constructed by researchers at the University of Nebraska–Lincoln (UNL), who established face and content validity through a review by six panel members familiar with both 4-H youth development and the 4-H Health Rocks! curriculum (Yan & de Guzman, 2011). All questions were written to elicit participants' self-reports of their statuses after and before participating in the program; the instrument consisted of 13 Likert scale questions, with the scale being 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree (Yan & de Guzman, 2011). The pilot test group consisted of 63 survey respondents, and principal component analysis showed three constructs, which the researchers identified as knowledge (four items), skills (four items), and assets (five items) (Yan & de Guzman, 2011). The researchers calculated Cronbach's alpha reliability values for the instrument's constructs and reported that those values for the overall before and after scales were above .70, which is considered acceptable (Bland & Altman, 1997); see Table 1. The instrument also included nine demographics questions regarding gender, race, ethnicity, and grade in school (Yan & de Guzman, 2011).

Table 1.
Cronbach's Alpha from 2010–2011 University of Nebraska Pilot Test of 4-H Health Rocks! Evaluation Instrument

Group (number of items) Cronbach's α: Before Cronbach's α: After
Knowledge (4 items) .755 .645
Skills (4 items) .769 .640
Assets (5 items) .864 .772
Total (13 items) .913 .831

During the 2011–2012 academic year, the 4-H Health Rocks! curriculum was implemented in Tennessee. The study reported here was approved by the institutional review boards at the University of Tennessee, Knoxville (study number 828913) and University of Tennessee at Martin (study number 15-383-E05-4007). According to Donaldson and Crowe (2013), the statewide 4-H Health Rocks! effort in Tennessee for the 2011–2012 academic year reached 18,850 youths with at least 10 hr of instruction in 17 counties. Given the large population of participants, intact groups totaling 100 youths per county were randomly selected and invited to complete a paper copy of the survey. For example, names of 4-H classes in a given county were randomly drawn and invited as intact groups of 4-H'ers to participate in the survey. The survey had a total of 1,192 respondents, for a response rate of 70% (1,192 of 1,700). The survey data were analyzed through the use of NCSS (Version 10). The data did not represent a normal distribution, and were positively skewed. A nonparametric test, the Wilcoxon signed-rank test, was implemented because the data were skewed and the goal of the analysis was to compare categorical data (Hollingsworth, Collins, Smith, & Nelson, 2011). The Wilcoxon signed-rank test is comparable to the paired-samples t-test, and it was used to compare participants' "before," or "retrospective-pre," and "after" or "post" responses. The Wilcoxon signed-rank test has been used to evaluate Extension risk management (Nagler, Bastian, Hewlett, & Weigel, 2007), food safety (Mathiasen, Morley, Chapman, & Powell, 2012), parenting and youth development (Behnke & Kelly, 2011), and 4-H Camp (Garton, Miltenberger, & Pruett, 2007) programs. Because the Wilcoxon signed-rank test is a nonparametric test, results apply only to the survey respondents rather than the larger population. A significance level of p ≤ .05 was established a priori.

Findings

The survey involved a total of 1,192 respondents. Not every question was answered by every respondent.

Objective One: Describing the Demographic Characteristics of Respondents

Respondents were asked to indicate their gender, race, ethnicity, grade in school, and age. More than half of the respondents were girls (52.43%, 625), a majority of respondents were Caucasian/White (65.02%, 775), and most were not Hispanic/Latino (85.40%, 1,018). Table 2 provides respondent demographic data.

Table 2.
Demographics of 4-H Health Rocks! Program Survey Respondents

Characteristic f %
Gender
Female 625 52.43
Male 545 45.72
Not reported 22 1.85
Race
Caucasian/White 775 65.02
African American/Black 190 15.94
Native American 40 3.36
Asian 14 1.17
Multiracial 75 6.29
Not reported 98 8.22
Ethnicity
Hispanic/Latino 72 6.04
Not Hispanic/Latino 1018 85.40
Unknown/not reported 102 8.56
Grade
4th grade 165 13.84
5th grade 252 21.14
6th grade 248 20.81
7th grade 283 23.74
8th grade 113 9.48
9th grade 60 5.03
10th grade and above 51 4.28
Not reported 20 1.68
Age
8 years old and younger 24 2.01
9 years old 82 6.88
10 years old 161 13.51
11 years old 266 22.32
12 years old 261 21.90
13 years old 153 12.84
14 years old 80 6.71
15 years old 38 3.19
16 years old and older 33 2.77
Not reported 94 7.89

Objective Two: Describing Differences, if Any, in the Post and Retrospective-Pre Scores of Respondents Regarding Knowledge, Skills, and Assets

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their knowledge levels before and after participation in the 4-H Health Rocks! program and involved four knowledge statements. With the Wilcoxon signed-rank test as applied in this context,

  • negative ranks indicated higher after (post) scores than before (retrospective-pre) scores,
  • positive ranks indicated lower after (post) scores than before (retrospective-pre) scores, and
  • ties indicated equal after (post) scores and before (retrospective-pre) scores.

Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the four knowledge items, as shown in Table 3.

Table 3.
Wilcoxon Signed-Rank Test Results for Respondent Knowledge Levels Before and After Participation in 4-H Health Rocks! Program

Item No. of respondents Mean rank Z p-value (2-tailed)
People who smoke can die of lung cancer. 1,114
Negative ranks 365 220.24 13.258 .000***
Positive ranks 70 206.33
Ties 679
Once you start smoking, it is hard to stop. 1,127
Negative ranks 432 264.68 13.829 .000***
Positive ranks 92 252.28
Ties 603
People who use drugs sometimes see or hear things that are not really there. 1,111
Negative ranks 468 291.15 14.723 .000***
Positive ranks 102 259.56
Ties 541
Using drugs can ruin my relationship with my family and friends. 1,120
Negative ranks 400 238.62 13.502 .000***
Positive ranks 76 237.89
Ties 644
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores.
***p ≤ .001.

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their skill levels before and after participation in the 4-H Health Rocks! program and involved four statements about skills. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the four skill items, as shown in Table 4.

Table 4.
Wilcoxon Signed-Rank Test Results for Respondent Skill Levels Before and After Participation in 4-H Health Rocks! Program

Item No. of respondents Mean rank Z p-value (2-tailed)
If a friend wants to try drugs, I can talk them out of it. 1,109
Negative ranks 449 293.03 13.156 .000***
Positive ranks 124 265.17
Ties 536
When I feel stressed, I am able to talk about it with people I trust. 1,112
Negative ranks 432 264.68 12.778 .000***
Positive ranks 92 252.28
Ties 603
I don't have to drink or smoke even if some other young people do it. 1,120
Negative ranks 468 291.15 12.247 .000***
Positive ranks 102 259.56
Ties 541
I am able to say "no" if others offered me drugs. 1,121
Negative ranks 400 238.62 12.368 .000***
Positive ranks 76 237.89
Ties 644
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores.
***p ≤ .001.

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their assets before and after participation in the 4-H Health Rocks! program and involved four statements about assets. All post scores were significantly higher than retrospective-pre scores, as shown in Table 5. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the five asset items, as shown in Table 5.

Table 5.
Wilcoxon Signed-Rank Test Results for Respondent Assets Before and After Participation in 4-H Health Rocks! Program

Item No. of respondents Mean rank Z p-value (2-tailed)
It is important for me to stay focused on learning at school. 1,122
Negative ranks 255 161.40 10.760 .000***
Positive ranks 59 140.63
Ties 808
I need to think about how my choices will affect my future. 1,117
Negative ranks 373 223.51 13.733 .000***
Positive ranks 69 210.62
Ties 675
I have goals for myself. 1,115
Negative ranks 299 173.79 12.744 .000***
Positive ranks 45 163.90
Ties 771
I feel good about myself. 1,115
Negative ranks 308 181.45 13.263 .000***
Positive ranks 47 155.37
Ties 760
I would help other kids like me to stay away from alcohol or other drugs. 1,123
Negative ranks 318 195.89 13.271 .000***
Positive ranks 59 151.84
Ties 746
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores.
***p ≤ .001.

Conclusions, Discussion, and Recommendations

The 4-H Health Rocks! program had positive, significant impacts on many respondents' self-perceived knowledge, skills, and assets regarding substance abuse prevention, as shown by retrospective post-then-pre survey results. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for all survey items. Additional data analysis should be undertaken for the purpose of understanding differences, if any exist, between and among groups of respondents, such as between female and male respondents or among respondents in different grades.

In spite of the evidence of positive, significant impacts, however, the Wilcoxon signed-rank test also showed that for all 13 items, the majority of youths reported no change from before to after participation in the program. This is an interesting finding that precipitates discussion and recommendations for both programming and evaluation. Because of youths' relatively high retrospective-pre scores, it appears that the majority of youths possessed the measured knowledge, skills, and assets before participating in the program. However, the program may be highly valuable in other ways. For example, although the majority of youths may possess the targeted skills, the program provides an important opportunity for youths to practice those skills. Also, future program evaluators should consider a different instrument for measuring program effectiveness. With the instrument used in the study described here, the forced-choice responses (1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree) may be too narrow to measure changes in youths' knowledge, skills, and assets.

The limitations of the study reported here should be noted and addressed in future research; most notable are limitations related to program delivery. We could not align specific delivery settings, such as residential camps, afterschool programs, or in-school 4-H club meetings, to respondents' perceptions; nor could we align specific lessons from the curriculum to respondents' perceptions. Also, we suggest that research related to ascertaining the appropriateness of the curriculum for different age groups is warranted. And, finally, although there is value in retrospective evaluation, the retrospective post-then-pre design was a limitation of the research described here. Future research should examine specific practices for conducting post-then-pre surveys with young people, given the limitations of retrospective post-then-pre questionnaires described by Klatt and Taylor-Powell (2005).

We recommend the formation of focus groups to provide further, more in-depth insight into the perceptions of 4-H Health Rocks! participants regarding the content of the program and ways to enhance the curriculum and instruction. This qualitative approach would complement the data already collected and provide a more comprehensive evaluative view of the 4-H Health Rocks! program. Additionally, it is recommended that Extension professionals consider random selection of participants for evaluation of other large-scale, statewide Extension programs.

Acknowledgments

We wish to express appreciation to the individuals and institutions that made our study possible: Dr. Yan Xia and Dr. Maria de Guzman of the University of Nebraska–Lincoln, who served as primary authors of the survey and conducted all validity and reliability studies; Justin Crowe, who coordinated the 4-H Health Rocks! implementation in Tennessee; and all the local Extension agents and volunteers who conducted the 4-H Health Rocks! program. 4-H Health Rocks! was funded by National 4-H Council.

References

Behnke, A. O., & Kelly, C. (2011). Creating programs to help Latino youth thrive at school: The influence of Latino parent involvement programs. Journal of Extension, 49(1), Article 1FEA7. Available at: https://www.joe.org/joe/2011february/a7.php

Bland, J., & Altman, D. (1997). Statistics notes: Cronbach's alpha. British Medical Journal, 314, 275.

Davis, G. A. (2003). Using a retrospective pre-post questionnaire to determine program impact. Journal of Extension, 41(4), Article 4TOT4. Available at: http://www.joe.org/joe/2003august/tt4.php

Donaldson, J. L., & Crowe, J. E. (2013). Healthy decisions. Retrieved from https://extension.tennessee.edu/publications/Documents/PB1811.pdf

Garton, M. S., Miltenberger, M., & Pruett, B. (2007). Does 4-H Camp influence life skill and leadership development? Journal of Extension, 45(4), Article 4FEA4. Available at: https://joe.org/joe/2007august/a4.php

Hollingsworth, R. G., Collins, T. P., Smith, V. E., & Nelson, S. C. (2011). Simple statistics for correlating survey responses. Journal of Extension, 49(5), Article 5TOT7. Available at: https://www.joe.org/joe/2011october/tt7.php

Johnston, L. D., O'Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2013). Monitoring the future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor, MI: Institute for Social Research, University of Michigan.

Klatt, J., & Taylor-Powell, E. (2005). Using the retrospective post-then-pre design, quick tips #27. Retrieved from http://www.uwex.edu/ces/pdande/resources/index.html

Kumaran, M., Fogarty, K., Fung, W. M., & Terminello, A. (2015). Improving healthy living youth development program outreach in Extension: Lessons learned from the 4-H Health Rocks! program. Journal of Extension, 53(3), Article 3RIB5. Available at: http://www.joe.org/joe/2015june/rb5.php

Mathiasen, L., Morley, K., Chapman, B., & Powell, D. (2012). Using a training video to improve agricultural workers' knowledge of on-farm food safety. Journal of Extension, 50(1), Article 1FEA6. Available at: https://www.joe.org/joe/2012february/a6.php

Miller, T., & Hendrie, D. (2008). Substance abuse prevention dollars and cents: A cost-benefit analysis. Rockville, MD: U.S. Department of Health and Human Services.

Nagler, A., Bastian, C. T., Hewlett, J. P., & Weigel, R. R. (2007). Risk management for ag families: Evaluation of an integrated educational program for producers on the northern plains. Journal of Extension, 45(3), Article 3RIB3. Available at: https://www.joe.org/joe/2007june/rb3.php

National 4-H Council. (2009). 4-H Health Rocks! Intermediate level. Chevy Chase, MD: National 4-H Council.

National Council on Alcoholism and Drug Dependence Inc. (2015). Alcohol, drugs and youth. Retrieved from https://ncadd.org/for-youth/233-overview

National Institute on Drug Abuse. (2014, January). DrugFacts: Nationwide trends. Retrieved from: http://www.drugabuse.gov/publications/drugfacts/nationwide-trends

NCSS 10 Statistical Software [Computer software]. Kaysville, UT: NCSS.

Raidl, M., Johnson, S., Gardiner, K., Denham, M., Spain, K., Lanting, R., . . . Barron, K. (2004). Use retrospective surveys to obtain complete data sets and measure impact in Extension programs. Journal of Extension, 42(2), Article 2RIB2. Available at: http://www.joe.org/joe/2004april/rb2.php

Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension, 27(2), Article 2FEA5. Available at: http://www.joe.org/joe/1989summer/a5.php

Self, C., Morgan, A. C., Fuhrman, N. E., & Navarro, M. (2013). An evaluation of the 4-H Health Rocks! program: Implications for program improvement. Journal of Drug Education, 43(1), 49–63.

Yan, X., & de Guzman, M. (2011). [4-H health rocks survey pilot test]. Unpublished raw data.