Abstract

Methods

The proposed model includes five steps: 1) problem definition; 2) goal setting; 3) method options; 4) action plans; and 5) ongoing assessment. The Tulane Prevention Research Center (PRC) conducted an SB evaluation with 19 schools in Orleans Parish. The evaluation,  developed with the 5-step model,  led to multi-level school data collection: administrative,  nutrition service, students, and cafeteria environmental scans.

Results 

Important practical data about lunchroom SBs was obtained. The evaluation, using the model, successfully demonstrated that all methods were achievable, participation reached expected levels, and identified goals were attained.

Application to Child Nutrition Professionals

Previously identified problems (safety, extra labor, cost, storage) with SB use make SB evaluation a high priority for school food management.  This proposed easy-to-use model can be applied by professionals and non-professionals in evaluating any school-based activities.  The demonstration of this SB evaluation showed that the model can provide efficient guidance that yields practical data for school decisions about SBs in school lunchrooms. School personnel will find this model easy to use; the steps of the model are transferable, and the model could potentially be applied to other evaluation issues.

 

Full Article

Program evaluation is a generalized term referring to any systematic method for collecting, analyzing, and using information to answer questions about projects, programs, and policies. Program evaluation has progressed through at least four generations of development: 1) the technical generation with computerization; 2) focus on goals and objectives as evaluation basis; 3) evaluation required to respond to needs of others (responsiveness generation); and 4) meta-evaluation (Issel, 2009).  Evaluation models have increased over time (Issel, 2009; Rowitz, 2001) through generations of increasing complexity with program designs and statistical methods becoming equally more complex (Guttmacher, Kelly, & Ruiz-Janecko, 2010).  Evaluation models have also become more specialized for defined areas of evaluation such as economics, research, public health programming, training and leadership, and medical procedures.

Complex models almost assured the need for experts to implement the models and provide evaluations based on statistical methods.  What is needed in many settings, such as schools, community organizations, and non-profit organizations, is a focus on simplified and easy-to-use evaluation methods, a skill that is transferable across multiple applications, as well as a simplified, efficient model that can function as a guide for evaluating in-house activities, programs, and procedures.

A very useful evaluation dichotomy is that between “outcome-focused” evaluations and “utilization-focused” evaluations (Issel, 2009).  The emergence of these “lines” of evaluation can account for much of the variation related to program evaluation (Issel, 2009). The purpose of outcome-focused evaluations is to show a program’s effect (2nd generation) while the purpose for utilization-focused evaluations is to provide information to stakeholders for decision-making and future program development (3rd generation) (Patton, 1997).

The purpose of this report is to present a simplified, efficient five-step “utilization-focused” evaluation model and demonstrate its applicability to evaluation of school salad bars (SBs).

School Salad Bars

Childhood obesity is a severe public health problem in the nation (Must et al., 1999; Polednak, 2008), as well as in Louisiana where about 57% of children and adolescents ages 10-17 years were reported as overweight or obese in 2007 (Singh, Kogan, & van Dyck, 2010).  A dietary strategy for preventing/reducing child weight is increasing fresh fruit and vegetable (FFV) consumption (Fung et al., 2012; Van Duyn & Pivonka, 2000).

Schools can be an effective venue for influencing the dietary patterns of children because of the time children spend in school and the large proportion of daily food consumed there (Story, Kaphingst, & French, 2006).  In 2014, the National School Lunch Program (NSLP) operated in over 99,000 public and non-profit private schools (K-12) and residential child care institutions, and provided low-cost or free lunches to more than 30.3 million daily (U. S. Department of Agriculture [USDA], 2016). With over 55 million youth enrolled in U.S. public and private primary and secondary schools, a school-based nutrition intervention strategy has the potential to reach the majority of the nation’s school-age children (Digest of Education Statistics, 2010).

The Centers for Disease Control and Prevention (CDC) (1996) established guidelines for school health programs to promote healthy eating and recommended an integration of school nutrition service and nutrition education, along with training for all school staff (CDC, 1996).  The report Making It Happen! (USDA Team Nutrition, 2005) suggested schools make healthful foods and beverages available, adopt marketing techniques that promote healthful choices, and limit student access to competitive foods.  Similar recommendations were made by the New Orleans Food Policy Advisory Committee, a committee originated by the Tulane Prevention Research Center (PRC), in its 2010 report Stepping up to the Plate: Transforming School Food in New Orleans (New Orleans Food Policy Advisory Committee, 2010).

Best practices in FFV consumption in the school environment now often include student access to salad bars (SBs) during lunch (French & Wechsler, 2004).  Though research is limited in this area, studies in Alabama (Reynolds, Franklin, & Binkley, 2000) and California (Joshi, Asuma, & Feenstra, 2008) found increased FFV consumption when SBs were introduced into the lunch period (Slusser, Cumberland, Browdy, Lange, & Neumann, 2007).  Crucial determinants for increasing FFV consumption from SBs included a variety of foods offered at the SB (Adams, Pelletier, Zive, & Sallis, 2005) and nutrition education at the school level (Suarez-Balcazar et al., 2007).

The Let’s Move Salad Bars to Schools is a national program with a goal that every child across the nation has daily access to FFV (Harris, et al., 2012; Let’s Move Salad Bars to Schools, 2016). About 4,800 SBS have been granted to schools throughout the country, and SB units (i.e., structures that stand alone and display and serve salad items) were granted to some receptive New Orleans schools, providing the Tulane PRC with the opportunity to evaluate the lunchroom SBs.  This paper describes the application of a problem-solving/decision-making model to evaluate the use of lunchroom salad bars.

METHODS:  Description of the Salad Bar Evaluation Model

The SB evaluation is presented as a demonstration of the application of the proposed evaluation model. The evaluation involved 19 schools that received SB units that stand alone and display and service food items. The design of the evaluation study was cross-sectional with data collection at multiple levels within the participating schools: administrative, school nutrition, students, and environmental scans which included cafeteria observations.  The model provided simple and efficient guidance for the evaluation that yielded important practical data for school decisions about the use of SBs as an alternate or supplement to the lunch main food line.

The model presented is an amalgam of three models posited by Liebler, Levine, and Rothman (1992), Robbins and Coulter (1999), and Rowitz (2001).  The resulting model, which was developed by the PRC SB evaluation committee (principal investigator, biostatistician, program coordinator, and one MPH investigator), involves a five-step approach: 1) problem definition; 2) goal setting; 3) method options; 4) action plans; and 5) ongoing assessment (Figure 1).  The following five steps were applied to the SB evaluation.

  1. Define the Problem. Use of SBs has been attempted numerous times in Orleans Parish school cafeterias with inconsistent and mixed effects, resulting in ineffective, inappropriate or discontinued use. A potential valuable strategy for increasing student consumption of FFV, therefore, has been under-utilized.  A utilization-focused evaluation model could provide the information required for informed decision-making by school personnel.

 

  1. Develop an Overall Goal. Consistent with a utilization-focused model (Patton, 1997), the evaluation committee decided that the goal would be to collect data relative to SB use, function and effectiveness. For these data to be useful to stakeholders, dissemination internally to the schools’ administrators and school nutrition personnel would be required. Dissemination externally to district and state level educators, administrators and school nutrition personnel, and legislators would also need to be included in the plan.

 

  1. Consider Options and Select Methods. Various options for data collection were considered. a) Qualitative or Quantitative Methods? In the decision-making process, it was determined that quantitative data collection would best serve the needs of this particular study.  Although qualitative methods, such as interviews and focus groups, would be a rich source of data usually not forthcoming with quantitative methods, qualitative methods would be time-consuming, require more resources to implement and evaluate, and may not be feasible during the school day. Student time is limited due to academic class periods, and school nutrition staff time is also limited because most work part-time.

 

  1. b) Survey Consideration. In considering surveys, the decision was between published and validated measurement instruments versus surveys developed specifically for this evaluation. A literature search revealed published studies on SB and other school-based meal service evaluations; however, none provided all of the questions that would sufficiently address the goals for this evaluation.

 

  1. c) Direct Observation. The evaluation committee decided to use direct observation to capture information about some crucial determinants such as nutrition education materials posted within the schools and marketing of healthy lunch items, particularly within the cafeteria, and photographic observation of the SBs at lunch periods. This direct observation was accomplished with checklists involving quantitative data collection.

 

  1. d) Groups to Target for Data Collection. The overall goal guided the decision-making relative to targeted groups for collecting data. Student surveys were needed to determine use, and data from food nutrition personnel and school administrators would provide information about SB function and support as well as effectiveness relative to implementation and cost.  Teacher surveys were considered, but previous studies by the evaluation committee had shown that teachers rarely eat in the student cafeteria, decreasing the probability that useful data would be obtained from this group; therefore, no teacher surveys were planned.

 

  1. Action Plans. Action plans were based on the above deliberations of the evaluation committee and their outcomes: a) use quantitative methods plus direct observation; b) develop surveys for this study using published questions plus new questions for additional variables; c) implement the surveys with students, administrators, and school nutrition personnel (director and staff), and d) conduct a school environmental scan which included cafeteria observations to capture the nutrition/food environment and the actual SB offerings during lunch periods.

 

  1. Ongoing Assessment. Usually problems arise during implementation that require flexibility. In the SB evaluation, field problems consistently arose that required flexibility in measurement and/or re-negotiations with school personnel. One problem, for example, was that enrollment data provided by the schools overestimated student enrollment and resulted in smaller sample sizes than originally expected.  A re-evaluation led to the recruitment of additional students in the grade levels of interest so that appropriate analyses could be performed.  While evaluation is usually viewed as progressing in a linear fashion, it is actually a cyclical process (Figure 1), making this final step important in the overall effectiveness of the model. Another unexpected situation arose when it was found that 7 schools that reportedly had SB units either did not have them or were not using them. This situation required an adjustment of the evaluation for those 7 schools and modified versions of the surveys were used deleting those items pertaining to the SBs. 

RESULTS

Table 1 lists the measurement targets and variables of interest that were included in the SB evaluation as a result  of the application of the problem-solving/decision-making model.  The study was conducted with 19 schools that had received SBs.  During implementation, it was found that 7 of these schools were not using the SB units, and so 12 schools with SB units received the full evaluation. The 7 schools without SB units received a modified evaluation in which questions relating to the SB were deleted. Some 1,200 students were targeted but more students (n=1,291) agreed to participate. One administrator and one school nutrition director from each school were targeted and surveyed.  Two environmental scans (direct observation) and two cafeteria observations were conducted in each school as intended.  There was a total of 37 school nutrition staff surveys across 12 schools.

Results of this evaluation study have been presented elsewhere (Andersen, Myers, O’Malley, Rose, & Johnson, 2016; Spruance, Myers, O’Malley, Rose, & Johnson, 2016). A brief summary of those results is provided.  Administrators in SB schools cited more facilitators, items that support SB use, than barriers compared to administrators in non-SB schools. Strong support from school nutrition staff and adequate equipment contributed to the successful implementation of the SBs. Regarding student use, it was found that females and students with healthier food preferences had higher odds of using the SB than male students and students reporting less healthy food preferences. Among secondary students, encouragement by students toward others for healthy eating and school-based SB marketing were significantly correlated with SB use.

  1. Table 1. Data Collection Methods for Evaluation of School-Based Salad Bars in New Orleans Schools
    Measurement School Category Main Variables of Interest
    Administrator Survey SB Schools SB background, structure, sustainability, budget, marketing/promotions, school wellness policy, nutrition education, other nutrition programming
    Administrator

    Survey-Modified

    Non-SB Schools SB background, potential sustainability, potential marketing/promotion, wellness policy, nutrition education, other nutrition programming
    School Nutrition Director

    Survey

    SB School SB background, maintenance, infrastructure, implementation, content, food procurement, staff training
    School Nutrition Director Survey-Modified Non-SB Schools SB background, plans for maintenance & infrastructure, future implementation, content, procurement, future staff training
    School Nutrition Staff Survey SB Schools Attitudes toward SB

    spacing

    Student Survey SB Schools Demographic information, SB use, food preferences, nutrition knowledge, social support
    School Environmental Scan

    Direct Observation

    SB Schools Vending machines, nutrition policies, marketing; cafeteria environment; SB characteristics, beverages available in cafeteria, a la carte line
    Observation of SB & Main Food Line SB Schools Student counts by gender & age category using SB and main food line
    Note: n = 12 Salad Bar (SB) Schools; n = 7 non-Salad Bar Schools

     

APPLICATIONS FOR CHILD NUTRITION PROFESSIONALS

Summary

The most feasible evaluation methods that do not considerably burden school resources are those that are uncomplicated, convenient and easily implemented.  This SB evaluation study using the problem-solving/decision-making model was successful in that all methods were achievable, participation reached expected levels, identified goals were reached, and plans for dissemination were activated. Only descriptive and non-parametric analyses were used; therefore, the number of participating schools was sufficient for the purposes of this evaluation.  Notable is that school participation was achieved without compensation because the budget was minimal.

The fifth and final model component, ongoing assessment, was important not only because of the cyclical nature of evaluation, but also because it allowed for the identification and resolution of challenges that arose during implementation.  Issues that arose during the SB evaluation, e.g. lower student enrollment than anticipated, and 7 schools not using the SB, demonstrated the value of an ongoing process of problem-resolution/decision-making and flexibility in implementation.

This paper describes the use of a model in developing and implementing an evaluation of school SBs.  An efficient process for evaluating school-based programs such as SBs can provide evidence-based data for practical use by school districts, school nutrition personnel and health behavior professionals, and can be accomplished in-house without external help.  The five-step model resulted in data collection from multiple levels within the schools.  Ongoing assessment was useful in resolving unexpected challenges.  Components of this problem-solving model are not new, but the blending of three previously published models along with application to the evaluation of school SBs are unique.  The CDC has provided a document, Framework for Program Evaluation in Public Health (CDC, 1999), which includes and goes beyond many of the areas discussed in this paper.  The terminology in the CDC document may be different from the model proposed here, but the major components are consistent with those proposed in this model.

Model Limitations and Strengths

A limitation of the presented model is that it does not include a preliminary decision-making process about whether or not to conduct the evaluation or study, which is included in many other models (e.g. Durch, Bailey, & Soto, 1997; Green, 1992; National Association of City and County Health Officers, 2000). In this case, that decision had already been made and was preemptive.  Also, the model, as presented, applies to a cross-sectional study design with almost all data self-reported.  Other than the direct observations in the cafeteria and the school environmental scans, all self-report data are subject to the well-known biases for self-report.

Strength is the simplicity and efficient application of the problem-solving/decision-making model that can enhance the work of an evaluation team and has the potential to provide guidance for evaluating other school-based activities.  This report can also serve as an instructional tool for early investigators, public health professionals, and school personnel for the collection of data pertaining to school lunch rooms.

ACKNOWLEDGEMENTS

This publication was supported by Cooperative Agreement #U48DP005050, under the Tulane Health Promotion and Disease Prevention Research Centers Program, funded by the Centers for Disease Control and Prevention.  The contents of this article are those of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention, nor of the U.S. Department of Health and Human Services.

References

Adams, M. A., Pelletier, R. L., Zive, M. M., & Sallis, J. F. (2005). Salad bars and fruit and vegetable consumption in elementary schools: A plate waste study. Journal of the American Dietetic Association, 105(11), 1789-1792. doi:10.1016/j.jada.2005.08.013

Andersen, L., Myers, L., O’Malley, K., Rose, D., & Johnson, C. C. (2016). Administrators and food service perspectives related to salad bar use. Journal of Child Nutrition & Management, 40(2).  Retrieved from https://schoolnutrition.org/JCNM/

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. Morbidity & Mortality Weekly Report, 48(RR-11), 1-40.

Centers for Disease Control and Prevention. (1996). Guidelines for school health programs to promote lifelong healthy eating. Morbidity & Mortality Weekly Report, 45(RR-9), 1-33.

Digest of Education Statistics 2010. (2011). Retrieved from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2011015

Durch, J. S., Bailey, L. A., & Stoto, M.A. (Eds.). (1997). Improving health in the community: A role for performance monitoring. Washington, DC: National Academy Press, Institute of Medicine.

French, S. A., & Wechsler, H. (2004). School-based research and initiatives: Fruit and vegetable environment, policy, and pricing workshop. Preventive Medicine, 39, 101-107.

doi:10.1016/j.ypmed.2003.10.007

Fung, C., Kuhle, S., Lu, C., Purcell, M., Schwartz, M., Story, K., & Veugelers, P. J. (2012). From “best practice” to “next practice”: The effectiveness of school-based health promotion in improving healthy eating and physical activity and preventing childhood obesity. International Journal of Behavioral Nutrition & Physical Activity, 9(1), 27. doi:10.1186/1479-5868-9-27

Green, L. W. (1992). PATCH: CDC’s Planned Approach to Community Health, an Application of PRECEED and an Inspiration for PROCEED. Journal of Health Education, 23(3), 140-147.

Guttmacher, S., Kelly, P. J., & Ruiz-Janecko, Y. (2010). Chapter 10: Evaluating a community-based intervention. Community-based health interventions. San Francisco, CA: John Wiley & Sons, Inc. (Jossey-Bass).

Harris, D. M., Seymour, J., Grummer-Strawn, L., Cooper, A., Collins, B., DiSogra, L., …Evans, N. (2012). Let’s Move Salad Bars to Schools: A public-private partnership to increase student fruit and vegetable consumption. Childhood Obesity, 8(4), 294-297. doi:10.1089/chi.2012.0094

Issel, L. M. (2009). Health program planning and evaluation: A practical, systematic approach for community health. Boston, MA: Jones and Bartlett.

Joshi, A., Asuma, A. M., & Feenstra, G. (2008). Do farm-to-school programs made a difference?  Findings and future research needs. Journal of Hunger & Environmental Nutrition, 3(2/3), 229-246. doi:10.1080/19320248.2012.703522

Let’s Move Salad Bars to Schools (2016). Retrieved from http://www.saladbars2schools.org/

Liebler, J. A., Levine, R. E., & Rothman, J. (1992). Management principles for health professionals (2nd ed.). Gaithersburg, MD: Aspen Publishers.

Must, A., Spadano, J., Coakley, E. H., Field, A. E., Colditz, G., & Dietz, W. H. (1999). The disease burden associated with overweight and obesity. Journal of the American Medical Association, 282(16), 1523-1529. doi:10.1001/jama.282.16.1523

National Association of City and County Health Officers. (2000). Protocol for assessing community excellence in environmental health (PACE-EH): A guidebook for local health officials. Washington, DC: National Association of County & City Health Officials.

New Orleans Food Policy Advisory Committee. (2010). Stepping up to the plate: Transforming school food in New Orleans. Retrieved from http://prc.tulane.edu/uploads/Transforming_School_Food.25891704-1319847670.pdf.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed). Thousand

Oaks, CA: Sage Publications.

Polednak, A. P. (2008). Estimating the number of US incident cancers attributable to obesity and the impact on temporal trends in incidence rates for obesity-related cancers. Cancer Detection & Prevention, 32(3), 190-199. doi:http://dx.doi.org/10.1016/j.cdp.2008.08.004

Reynolds, K. D., Franklin, F. A., Binkley, D., Raczynski, J. M., Harrington, K. F., Kirk, K. A., & Person, S. (2000). Increasing the fruit and vegetable consumption of fourth-graders: Results from the High 5 project. Preventive Medicine, 30(4), 309-319. doi:10.1006/pmed.1999.0630

Robbins, S. P. & Coulter, M. (1999). Management (6th ed.). New York, NY: Prentice Hall.

Rowitz, L. (2001). Public health leadership: Putting principles into practice. Gaithersburg, MD: Aspen Publishers.

Singh, G. K., Kogan, M. D., & van Dyck, P. C. (2010). Changes in state-specific childhood obesity and overweight prevalence in the United States from 2003 to 2007. Archives of Pediatrics & Adolescent Medicine, 164(7), 598-607. doi:10.1001/archpediatrics.2010.84

Slusser, W. M., Cumberland, W. G., Browdy, B. L., Lange, L., & Neumann, C. (2007). A school salad bar increases frequency of fruit and vegetable consumption among children living in low-income households. Public Health Nutrition, 10(12), 1490-1496.

doi:10.1017/S1368980007000444

Spruance, L., Myers, L., O’Malley, K., Rose, D., & Johnson, C. C. (in press) Individual- and school-level factors related to school-based salad bar use among children and adolescents. Health Education & Behavior.

Story, M., Kaphingst, K. M., & French, S. (2006). The role of schools in obesity prevention. The Future of Children, 16(1), 109-142. doi:10.1353/foc.2006.0007

Suarez-Balcazar, Y., Redmond, L., Kouba, J., Hellwig, M., Davis, R., Martinez, L. I., & Jones, L. (2007). Introducing systems change in the schools: The case of school luncheons and vending machines. American Journal of Community Psychology, 39, 335-345.

doi:10.1007/s10464-007-9102-7

U.S. Department of Agriculture, Economic Research Service. (2016). National School Lunch Program. Retrieved from https://www.ers.usda.gov/topics/food-nutrition-assistance/child-nutrition-programs/national-school-lunch-program.aspx

U.S. Department of Agriculture, Team Nutrition. (2005). Making it happen! School nutrition success stories. Retrieved from www.fns.usda.gov/tn/Resources/makingithappen.html

Van Duyn, M. A. S., & Pivonka, E. (2000). Overview of the health benefits of fruit and vegetable consumption for the dietetics professional: Selected literature. Journal of the American Dietetic Association, 100(12), 1511-1521. doi:10.1016/S0002-8223(00)00420-X

Biography

Johnson, O’Malley, Begalieva, and Myers are associated with the School of Public Health and Tropical Medicine at Tulane University in New Orleans, Louisiana.  Johnson is a Professor, O’Malley is Assistant Director, and Begalieva is a Clinical Associate Professor in the Department of Global Community Health and Behavioral Sciences at Tulane, while Myers is a Professor in the Department of Global Biostatistics and Data Science.  Spruance is an Assistant Professor in the Department of Health Science at Brigham Young University in Provo, Utah.

Purpose / Objectives

Evaluation of school-based activities is a high priority for school personnel.  Nutrition activities, such as salad bars (SBs) incorporated into school lunchrooms, may increase children’s consumption of low-energy, high fiber diets. The purpose of this paper is to describe a problem-solving/decision-making model and demonstrate its application in the evaluation of a school-based SB program.