Process Evaluation
Complete and Acceptable Delivery of The Health-Focused Mindset (HFM) Program
To qualify as a complete and acceptable delivery of The Health-Focused Mindset (HFM) Program, there should be four fundamental components: two environmental components creating both a school implementation team and a small group comprised of peers for the second educational module; two educational modules, The “Skinny” on Diet and a Social Media Health Promotion project.
Environmental Component
The Health-Focused Mindset (HFM) Program’s environmental component is twofold: a school-based implementation team and a small group comprised of students. At each participating high school, an implementation team will be created with stakeholders at the school. This team will function throughout the duration of both educational modules. At minimum, the school stakeholder team will consist of one administrator, two teachers, and additional related staff members. Ideally, the school environment will support the program by providing both content and verbal support to students as well as by providing resource support in the form of time, technology, and advertising of the program. Team members will help disseminate program staff-provided training videos and note sheets to school-based instructors. The small group will be made up of no more than 6 students and no less than 4 students. Each group will be randomly selected from a pool of peers, from all different grade levels.
Educational Component
The Health-Focused Mindset (HFM) Program includes two modules each lasting for 6-week intervals, meeting for about 30 to 45 minutes a week during the first module sessions and 1 to 1.5 hours a week for the second module sessions. Because the curriculum is administered through a secure online platform, extensive training is not necessary for school-based course administrators. In the second module, school-based administrators of the course will have a brief virtual call where program staff explains best practices and expectations of them during the module. While the second module is also web-based, school-based administrators will need to have a brief overview of facilitating group work amongst small groups as they work on their social media health promotion projects. If there are questions during the work sessions in module two, there will be a live chat option with an HFM program staff member during their allotted work time. Sessions for each module must be delivered to students and completed in order as they were designed to be completed in sequence, as each builds on the previous lessons.
The “Skinny” on Diet module will contain, at a minimum, the following:
- School-based instructors will provide computer delivery of curriculum lessons for each student using an individually-provided, unique user code.
- Instructors will ensure students complete web-based reflections and responses for each lesson.
- Instructors will provide take-home assignments so students can engage their families with the curriculum content.
The Social Media Health Promotion module will include:
- Student participants working in small groups to design and create a social media health promotion campaign about diet culture and nutrition diet.
- Participation from students in each stage of the process of building a social media campaign: brainstorming, storyboarding, design, and content production.
- Students will peer-review campaigns from other groups and provide constructed feedback on how well the brief was addressed and how accessible the content was.
- Social media campaigns will be shared on indicated or intended media platforms
Potential Process Evaluation Questions
| Target of Question | Process- Evaluation Question | Method of Assessment for Question | Resources Required |
| Activities | To what extent were all of the intended methods, strategies, and/or activities used? | Data would come from the web-based lesson delivery platform, both from completion statistics and from the quality of completion. | Data from the real-time backend database linked to the online program |
| Reach |
|
Data would be sourced from the schools participating in the HFM program, namely full class lists. These lists would then be compared to the completion data from the online platform using the unique log-in codes assigned to specific students. | Classroom rosters and data from the real-time backend database linked to the online program |
| Fidelity | To what extent was the intervention implemented consistently with the underlying theory and philosophy of the program? | A combination of reports from teachers assisting the implementation of the curriculum and HFM staff observation using a checklist of expected characteristics of implementation. Data will also come from the web-based system and the unique log-in codes for participants. Completion of lessons could be tracked by a real-time backend database linked to the online program. | A checklist of expected characteristics of implementation used for reports from teachers assisting the implementation of the curriculum and for HFM staff observation |
| Context | General
Environmental
|
Data would come from reflection surveys both for participants and school staff, teachers, and administrators involved in the program. The primary methods and tools are survey interviews using open-ended questions to address barriers to implementation and delivery, what time was allotted for the program each week, and specific examples of support given to participants. Survey responses would be cross-referenced between populations to check accuracy. | Reflection (posttest) surveys from both participants and school-based staff using open-ended questions |
| Program Management |
|
Sources and methods will include daily documentation from HFM program staff regarding all activities involved in identifying and recruiting administrators within schools and surveys from both the HFM school-based team and the HFM staff. | HFM documentation and surveys from both the HFM school-based team and the HFM staff |
| Dose Delivered |
|
Possible data sources and methods include the real-time backend database linked to the online program, reports from the school-based team, HFM staff observation. These methods will require a clear checklist of content covered and requested participant output measures. | Data from the real-time backend database linked to the online program, reports from the school-based team, HFM staff observation, and a clear checklist |
| Dose Received |
|
Possible data sources include teachers, staff, administrators, and student participants. Methods and tools include administering brief satisfaction scales, possibly Likert scale-based, and conducting interviews and/or focus groups with open-ended questions. | Likert scale-based satisfaction surveys and interviews using open-ended questions |
Summary of the Process Evaluation
After consideration, the most important areas of priority for process evaluation questions for The Health-Focused Mindset (HFM) Program are fidelity (quality), reach (participation rate), context, dose delivered (completeness), and dose received (exposure). Program management also deserves an honorable mention, which could be used in a case by case situation depending on the context of the school. However, context can serve as a placeholder, where it answers two main questions:
- What factors in the organization, community, social/political context, or other situational issues could potentially affect either intervention implementation or the intervention outcome?
- Did schools provide support to participating students?
Fidelity can be addressed in program implementation by asking the question: To what extent was the intervention implemented consistently with the underlying theory and philosophy of the program? Reach can best be addressed by asking a combination of questions:
- What proportion of the priority target audience participated in (attended) each program session? How many participated in at least one-half of possible sessions?
- How many students completed the social media promotion project? Of those who completed the project, how many shared the promotion project on social media?
While dose delivered can be internally addressed through the online platform, the second module of the program necessitates a question from both participants and school staff: To what extent were all of the intended methods, strategies, and/or activities used, especially in regards to the social support strategies of the second module? Dose received can be addressed in a posttest given to participants at the conclusion of the program as well as six months later. However, an important evaluation to ask of the interactive second module is: To what extent were students present at intervention activities engaged in the activities, especially in the social media promotion project?
In this unique program, the technologically-based curriculum dissemination tool helps simply evaluation data on the receiving end. However, to complete a process evaluation, checklists must be developed on the early end through the online platform. Interviews and surveys using open-ended questions and Likert scale-based questions are also crucial to the evaluation process.
References
McKenzie, J, Neiger, B., & Thackeray, R. (2017). Planning, Implementing, and Evaluating Health Promotion Programs: A Primer. 7th ed. Pearson Education, Inc.
Saunders, R.P., Evans, M. H., & Joshi, P. (2005). Developing a Process-Evaluation Plan for
Assessing Health Promotion Program Implementation: A How-To Guide. Health Promotion Practice.6: 134.
Schwinn, T., Schinke, S., Fang, L., & Kandasamy, S. (2013). A web-based, health promotion program for adolescent girls and their mothers who reside in public housing. Addictive Behaviors, 39(4), 757–760. https://doi.org/10.1016/j.addbeh.2013.11.029.