Mixed methods process evaluation of Generation SKILLZ among adolescents in South Africa

DETAILED SUMMARY

Importance

• South Africa is enduring the world’s most severe HIV epidemic, with approximately 30% of new infections occurring among 15- to 24-year-olds, and school-based prevention programs are seen as an important strategy for reaching this age group.
• Randomized controlled trials can show whether such programs work, but process evaluations are needed to understand how and why they do, or do not, achieve their intended effects, and how they might be improved.

Objective

• To assess the fidelity and quality of implementation, coach training and experiences, and student engagement with Generation Skillz Utshintsho, a split-sex HIV prevention curriculum, in order to help interpret the results of the GOAL Trial and inform future intervention adaptation.

Study details

• Study design: Mixed methods study (no control group)
• Data collection tools: Structured observations of intervention session delivery (n=49); unstructured observations of coach training; focus group discussions with coaches (n=3 FGDs; 25 coaches); focus group discussions with students (n=15 FGDs; 99 students)
• Participants: 25 Grassroot Soccer coaches and 99 secondary school students across two GOAL Trial sites, drawn from the larger GOAL Trial population of approximately 4,500 students
• Implementation setting: 23 intervention schools (8 in Cape Town, 15 in Port Elizabeth) in urban and peri-urban informal settlements, South Africa
• Study period: 2013–2014 (process evaluation conducted alongside delivery of Generation Skillz Utshintsho, March–September 2013)

Intervention

• Generation Skillz Utshintsho: A split-sex, discussion-based curriculum delivered by Grassroot Soccer Coaches as the second component of the combined Generation SKILLZ program within the GOAL Trial, using sport-based themes, activities, and metaphors to address multiple sexual partnerships, age-disparate relationships, and gender-based violence.

Results

• Observed practices were implemented with an average fidelity and quality score of 68% (range 38–85% across practices and sites); key messages were delivered 91.4% of the time, and session goals were judged to have been met 82.6% of the time.
• Female-only sessions were implemented with greater fidelity than male-only sessions in four of the five split-sex practices.
• Core curriculum components, including warm-up/cool-down activities, the curriculum contract, and the coach’s personal story, were frequently skipped, often because coaches found them confusing, impractical, or uncomfortable to deliver.
• Coaches had the materials needed to implement sessions in only 55% of observed practices, with a notable difference between Cape Town (84%) and Port Elizabeth (25%).
• Training quality was rated high overall, though gaps between training and implementation, and mid-course curriculum changes, were sources of frustration for coaches.
• Students generally valued the split-sex, discussion-based approach and found the topics relevant, but requested a longer program, more discussion time on certain topics, and more opportunities for dialogue between male and female students.

Conclusions and implications

• The GOAL Trial’s modest behavioral results may be explained, in part, by real-world limitations in how Generation Skillz Utshintsho was implemented, alongside gaps in understanding of how its components were intended to function.
• Strengthening fidelity to core curriculum components, ensuring consistent access to materials across sites, and extending opportunities for discussion may help enhance the effectiveness of similar sport-based HIV prevention curricula in future implementations.