Steps to Nutrition Success: Program Self-Assessment for Child Care Centers and Family Day Care Homes Participating in the Child and Adult Care Food Program

Please note that this study was published before the implementation of Healthy, Hunger-Free Kids Act of 2010, which went into effect during the 2012-13 school year, and its provision for Smart Snacks Nutrition Standards for Competitive Food in Schools, implemented during the 2014-15 school year. As such, certain research may not be relevant today.

Child care professionals in child care centers and Family Day Care Homes (FDCHs) strive to provide the best care possible through the application of various Child and Adult Care Food Program (CACFP) guidelines and other state and local guiding principles and resources. In a Continuous Program Improvement (CPI) cycle, the first step is self-assessment (Burkhalter, 1996; Hoffman & Herren, 2003). Self-assessment empowers professionals to determine current level of quality and identify strengths and limitations of their program.

Several studies have been directed at identifying students’ perceived benefits and barriers to participating in the SBP (Dixit, Houser, & Sampson, 1999; Reddan, Wahlstrom, & Reicks, 2002). However, there are many issues affecting SBP participation beyond students’ control. Short meal periods, unfavorable bus schedules, and social stigma have been identified as disadvantages and barriers to SBP participation (McDonnell, Probart, Weirich, Hartman, & Birkenshaw, 2004; Ragno, 1994). One of the primary stakeholders in the SBP is the school nutrition director (SND). The SND is the individual responsible for the implementation, maintenance, and financial viability of the program. However, only one recent investigation could be found that included SNDs’ perceptions regarding school nutrition programs and student participation in the SBP (McDonnell et al. 2004). McDonnell et al. reported that SNDs expressed frustration regarding limited time for the breakfast meal period and pre-class events interfering with SBP participation. SNDs also identified positive interactions between school nutrition staff and students as contributing to higher school meal participation. The objectives of this study were to further investigate the perceived advantages, disadvantages, and barriers to participation by elementary students in the SBP held by SNDs and teachers in low participation states.

CPI is a challenge affecting all facets of the CACFP. While providing high quality child care services, including nutrition and foodservice to children in their care, the child care provider participating in the CACFP must balance resources of time, staff, and finances. Finding the time and other resources needed to focus on self-improvement and to be able to demonstrate that children are receiving excellent service, although important, may be a challenge. However, self- assessment has strong theoretical and practical support in the educational research literature (Baldridge National Quality Program, 2006; Burkhalter, 1996; Ginsburg, Lauland, & Pane, 2005; Hoffman & Herren, 2003). Additionally, professional associations, institutions of higher education, and organizations offering certification, credentialing, or special recognition may integrate the self-assessment and continuous improvement process into their programs.

While self-assessments are important first steps in driving quality and performance, it should be noted that some researchers have identified limitations inherent in self-assessment techniques. Dunning, Heath, and Suls (2004) reviewed psychological research that demonstrates difficulties with interpretation of self-assessment instruments. For example, when respondents are completing self-assessments, there may be a tendency to overrate performance. Also, some respondents may initially overrate their performance but upon re-assessment, more appropriately rate their scores (Taylor, 2001). This may happen because, initially, respondents have limited experience and lack a frame of reference to guide their assessment of their skills and knowledge. Over time, respondents become more familiar with practices, procedures, and terminology; therefore, they adjust their responses based on this new or deeper understanding of the issues.

Yet, some respondents may underrate themselves. Dunning et al. noted that “people have a modest level of insight, at best, into their skill and character” (p. 71). These authors found a modest correlation between self-assessment and performance across a wide range of occupations and settings.

Cranton (2000) advocated when planning instruction/professional development for adults, one should start with an assessment of knowledge and experience. Often, formal tests and surveys “may be seen as intimidating and threatening to administer…prior to meeting learners or in the first session” (p. 26).

While difficulties associated with self-assessments are noted, it is believed that benefits of such instruments outweigh potential limitations. Organizations in the public and private sectors have developed a variety of self-assessment materials for determining quality child care. The purpose of this study was to develop and validate a program self-assessment instrument of identified best practices in the areas of nutrition and foodservice aspects to facilitate CPI for child care providers in centers and FDCHs participating in the CACFP. To accomplish this purpose, the following objectives were established:

• Identify all major CACFP program regulations and guidance materials as well as other appropriate child care guidance materials related to quality nutrition and foodservice in the “child care setting” (e.g., Head Start program performance standards).
• Determine the areas or aspects of quality nutrition and foodservice to be included in the program self-assessment instrument.
• Seek input from an expert panel on the content and format for the program self- assessment instrument.
• Assess the efficacy of the program self-assessment instrument using a review panel of CACFP providers in child care centers and FDCHs.

Methodology

Phase 1

Expert panel.

A select group of 10 CACFP professionals, who were subject matter experts, assisted the researchers in development of the draft program self-assessment instrument. The expert panel represented state agency administrators responsible for the CACFP in their states, a USDA nutritionist, and sponsoring organization directors. These participants represented five states, five USDA regions, USDA Food and Nutrition Services, and National Food Service Management Institute (NFSMI). The Mississippi State University Human Subjects Protection Review Committee approved the study protocol.

Instrument development.

Food and nutrition quality indicators were identified from the CACFP regulations related to the nutrition and foodservice offered in the child care setting (Child Nutrition and WIC Reauthorization Act of 1998), guidance materials, and other appropriate standards (American Dietetic Association, 1999a, 1999b; Code of Federal Regulations, 2001; Maternal and Child Health Bureau, 2002; U.S. Department of Agriculture [USDA], Food and Nutrition Service [FNS], 1999a, 1999b; U.S. Department of Health and Human Services, 1996). Based on the recommendations of the expert panel, these quality indicators were organized into content sections to form a program self-assessment instrument for child care centers and one for FDCHs.

The expert panel performed a content validation procedure on these quality indicators identified by the researchers. This procedure was twofold. The expert panel completed a preliminary checklist prior to attending a work group session. During a two-day work session, a group consensus procedure was used to provide feedback on the content, scope, and wording of the quality indicators. The basic organization and format for the program self-assessment instruments was determined.

In addition, the expert panel members identified several issues that were critical to the evaluation of the instruments and the self-assessment process. The expert panel members agreed that to be useful in the CPI process, the final instruments would need to be easy to use, and the CPI process recommended would be convenient for busy child care givers. The next step in developing the final instruments involved a review panel. The review panel would comment on the ease and convenience of the draft instruments as well as comment on how likely they would be to conduct program self-assessment on a regular basis using the instruments under development. Finally, the researchers and the expert panel were interested in determining if the review panel participants would think training on the use of the program self-assessment instruments would be helpful and whether some type of recognition for having completed the process would be important to them. The expert panel provided input to the researchers in naming the instruments the Steps to Nutrition Success Checklist…Child Care Centers and Steps to Nutrition Success Checklist…Family Day Care Homes.

 Phase 2

Review panel.

The expert panel participants provided names and contact information for 29 child care centers and 56 FDCH providers to review and complete the program self-assessment instruments. The researchers confirmed that these review panel respondents had access to all aspects of the nutrition and foodservice covered in the instruments. Review panel respondents received the appropriate version through the mail with instructions on how to complete it, as well as a self- addressed, postage-paid return envelope.

Checklist instrument evaluation.

Based on issues identified by the expert panel participants, researchers developed a tool for the review panel respondents to evaluate the purpose and procedures of the checklists. They determined that upon completion of the instruments, a review panel respondent would be asked to complete a 12-statement evaluation form. These statements would measure the value, usefulness, and clarity of both instruments. Even though child care and FDCH providers reviewed and completed different versions of the checklists, each would complete identical evaluation forms. The evaluation survey was mailed to the review panel respondents with their respective checklist. The review panel respondents were directed to return the evaluation survey, along with comments to the researchers.

Data analysis.

Responses to the evaluation were tabulated and analyzed. Descriptive statistics reported means and standard deviations. Means were compared and individual sample t tests were performed.

Results And Discussion

The purpose of this study was to develop and validate a program self-assessment instrument of identified best practices in the areas of nutrition and foodservice aspects to facilitate CPI for child care providers in centers and FDCHs participating in the CACFP. This kind of resource can assist child care providers with the appropriate documentation of their program practices for their sponsoring organization and other training and professional development organizations.

Steps to Nutrition Checklists

The researchers determined that program self-assessment was an appropriate first step for child care professionals to measure program operating practices. Child care providers operating child care centers and FDCHs differ in regulatory guidelines; because of the differences, the expert panel expressed a need to create separate instruments for child care center and FDCH providers. The instruments were to reflect quality indicators with language tailored for the child care program participating in the CACFP. The expert panel participants made the following recommendations with regard to the specific self-assessment process: create separate versions for child care center and FDCH providers; organize by content sections to remain consistent with CACFP guidance materials from the U.S. Department of Agriculture Food and Nutrition Service (USDA, FNS, 1999a, 1999b); refer to the quality indicators as best practices; make the instruments web-based; and name the instruments Steps to Nutrition Success Checklist…Child Care Center and Steps to Nutrition Success Checklist …Family Day Care Home.

Both checklists have identical directions for implementation of a five-step process to complete the self-assessment and include three sections: 1) Administration and Operations, 2) Nutrition, and 3) Health, Safety, and Well-Being of Children – Working with Parents and Others in the Community. Each section is divided into best practice checklists, which serve as a sub-scale or category within the section. The checklists are customized for use in child care centers and FDCHs. The differences in the checklists are presented by the content sections used to organize the checklists for the end user. Respondents are directed to check “Yes” if the program is adhering to the best practice, “Some” if the program has begun to work on it, “Maybe” if the program might work on it, and “NA” if the program determines the best practice is not needed. Figures 1 and 2 illustrate the format of Steps to Nutrition Checklist…Child Care Center and Steps to Nutrition Checklist…Family Day Care Homes, respectively.

Figure 1. Steps to Nutrition Success Checklist…Day Care Centers

Figure 2. Steps to Nutrition Success Checklist…Family Day Care Homes

The differences in the checklists are presented by the content sections used to organize the instruments for the end user (Table 1). Each checklist covered the same best practices but was worded differently to reflect these different environments as based on input from the expert panel. Feedback from the evaluation by the review panel provided valuable information on how the instruments may be used and how state agencies, sponsoring organizations, and the NFSMI might provide assistance to CACFP participants as they continue to maintain and improve program quality.

Table 2. Levels of agreementa on usefulness and value of the Steps to Nutrition Success Checklist as reported by review panel respondents (N=43).

Evaluation of Checklists

 Of 29 child care center and 56 FDCH providers who were mailed checklists, 12 child care center providers and 31 FDCH providers responded. This resulted in a return rate of 51%, (N = 43).

Respondents consisted of six independent child care center providers, six Head Start centers, and 31 FDCH providers.

Responses were tabulated by respondent groups, such as child care centers or FDCHs and then aggregated to get a total measurement for each group. Additionally, respondents could comment on or make suggested modifications for best practice quality indicators. While respondents provided written comments, they did not make specific suggestions for any modification to best practices. When respondents were asked if they participated in program self-assessment on a regular basis, 9 child care centers and 12 FDCH providers indicated they participated in program self-assessment.

Table 2 displays the means and standard deviations for each statement. The results indicated that 9 of the 12 statements had a mean score of 3.0 ± 0.9 or greater on the 4-point agreement scale.

Findings could be interpreted as support for the positive value and usefulness of the program self-assessment checklists. The highest mean scores were found on statements 1 and 2, both of which had a mean of 3.7 ± 0.7. This would suggest agreement that the checklists are consistent with CACFP implementation and regulations. The lowest mean scores were found on items 9 and 12, both of which had a mean of 2.9 ± 0.8 and 2.9 ± 0.9, respectively. Statement 9 was regarding the development of a plan for improvement based on the results of the checklist and statement 12 was interest for recognition by sponsoring agencies for completing it.

Visual inspection of the data suggests a difference between how the two groups viewed the readability and clarity (statements 1 – 5) and the utility (statements 7 – 10) of the checklists. Statements 1 through 5 had a mean score of 3.5 ± 0.7 to 3.7 ± 0.6, which suggests that respondents may have favored the organization and look of the checklists in contrast to the perceived usefulness. Still, the means of all those statements were a 3.0 ± 0.8 or above, with the exception of statement 9, suggesting agreement that the checklists were useful, an accurate representation of best practices, and valuable for program improvement. There was consensus with no meaningful differences in responses between child care center and FDCH providers on those statements.

Differences were noted between the study groups (p < .05) on statements 11 and 12. The FDCH group (3.1 ± 0.8) was more likely than the child care center group (2.6 ± 0.7) to embrace training on the checklist. In addition, the FDCH group (3.0 ± 1.0) may be more likely to embrace recognition by sponsoring agencies or state agencies for completing the checklists than the child care center group (2.6 ± 0.8). Examination of possible differences between these groups may be an area of interest for further research.

Data suggest that respondents valued the assessment of their programs when utilizing the checklists. Three of the 43 respondents commented that the time required to complete the program self-assessment was a factor considered when evaluating the checklists. All respondents recorded some comments about how well the child care programs operated, using various best practices identified in the program self-assessment instruments. They also agreed (3.1 ± 0.7) that they would be willing to participate in the assessment process on a regular basis, for instance, once a year.

Researchers were interested to know if respondents thought the checklist would assist them in CPI efforts. Statements 7 – 10 addressed the perceived usefulness of the checklist in helping respondents reach program requirements and goals, willingness to complete the checklist, willingness to develop a plan of improvement, and long-term impact on their program using the checklist. The means (3.3 ± 0.7, 3.1 ± 0.8, 2.9 ±0.8, and 3.1 ± 0.7, respectively) of these statements confirm the utility of the checklists. There was also a need to know if the checklists were congruent with review panel respondents’ understanding and practice of industry quality indicators. Statements 1 – 3 ask respondents if best practices are consistent with rules and regulations of the CACFP and if they represent best care. These three statements had the highest means, 3.7 ± 0.6, 3.7 ± 0.6, and 3.6 ± 0.6, respectively. These responses suggest that the checklists captured best practices and accurately represented them. Likewise, respondents seemed to think best practices were stated clearly and easy to apply, as indicated in by their responses in statements 4 (3.5 ± 0.7) and 5 (3.5 ± 0.8).

Table 1. Comparison of Best Practices Included in the Child Care and FDCH Steps to Nutrition Success Checklist

*p<0.05

Conclusions And Applications

A number of limitations can be identified in the present research. Demographic data such as age, gender, educational level, and years of experience in CACFP child care were not collected. In future studies, that information may be useful in explaining differences in responses between the study groups. The return rate represented approximately 51% of those contacted to participate in the review process. Thus, there may be a difference between the responders and non-responders. In addition, the review panel was a convenience sample and was not chosen at random. These factors may limit researchers’ ability to generalize these findings to the industry at large.

General limitations of self-assessments were previously described. While the checklists are not self-assessments of personal characteristics, difficulties described previously may logically apply to program assessment by program providers. Some respondents may overrate, underrate, or rate their programs inappropriately based on limited confidence or ability to objectively rate performance. If sponsoring agencies choose to use results in a punitive manner versus empowering child care providers to assess the program and link their assessment to a training plan, child care providers may overrate their program’s assessment. If turnover is high in a particular program, and new staff members are unfamiliar with terminology or best practices, performance may be systematically over- or underrated. The checklists minimize these effects by evaluating the progress and intent of behavior rather than making value judgments about quality of performance. Using a 4-point rating scale (yes, some, maybe, NA), for example, a child care provider rating their assessment of their program is forced to make a judgment or have no opinion on the quality of the work.

The checklists and the proposed program self-assessment process were not developed to replace any regulatory evaluation or monitoring required for CACFP participation by either study group. The intent was to provide CACFP professionals with an instrument to assess their programs, while following all federal, state, and local requirements. In doing so, the checklists would assist providers in establishing quality foodservice programs that serve children and that they are consistent with the quality care as outlined in the CACFP requirements and best practices promoted by the industry. This self-monitoring process provides a time to reflect and assess existing practices. When CACFP providers assess their program practices on a regular basis to industry’s best practices, they are actively participating in CPI. When modification in services or changes in practices are then made that move a program toward best practices and enhanced quality assurance, CPI is actively performed. As education agencies and organizations align with more research or evidence-based practices, these checklists can provide means of ongoing review and documentation of growth in the care given to children.

Assessment results can be benchmarked from one year to the next. As detailed previously, while respondents of this study indicated they would be willing to participate in an ongoing implementation of the instrument, they appeared to be slightly less likely to develop a plan for improvement. This may be an indication that child care providers do not feel confident in developing a program improvement plan from results of a program self-assessment. Further

investigation into this issue is warranted. The annual process for program improvement using the checklists could involve four stages: at the beginning of the year, providers could complete the appropriate checklist. From that completed checklist, they would write a program improvement plan targeting continuation of specific strengths and outlining intervention strategies for weaknesses of the program. Periodically throughout the year, providers could document progress of their strategies, and at the end of the year, providers would reassess the program. This reassessment would serve as the pre-assessment for the next year. In this way, the checklists serve as a pre-post resource and program improvement plan. Ongoing notations of success can serve as informal research and evidence of quality improvement. Training organizations may be able to use the checklists as part of establishing training needs and designing programming for child care professionals. Practical applications of the checklists include the development of education and training programs and other outreach services for child care professionals.

Additionally, child care sponsors could also implement a similar program improvement plan by state or region served. Results from annual checklists could be aggregated, and once tabulated, CACFP sponsors could determine which best practices had increased and which needed additional attention. This method of identification could be used to draw attention to strengths and help assist those responsible for training focus specifically on perceived training needs as identified through the assessment process.

The checklists were designed to empower child care providers and increase adherence to best practices through a non-threatening, yet systematic process. Child care providers have the best vantage point to assess their program and independently identify strengths and weaknesses of their own services. By doing so, they would be better informed to take ownership of their own weaknesses and adjust performance in an effort to lead to program improvement.

Written comments from respondents indicated they are presently engaged in various training and assessment activities that assist in maintaining and improving quality in their child care programs. Numerous comments suggest additional training is needed in several key areas, including dealing with choking, ideas for menus, and safe food handling practices. If sponsors aggregated the checklists results from their providers, statements that received a majority of “maybe” or “NA” could be incorporated into professional development opportunities.

Consequently, statements on the checklists can be converted directly to training objectives and competencies, assisting state and sponsoring agencies, and other support organizations for training program development. In addition to assisting the development of training, it would also serve as a tool to understand practitioners’ prior learning. We know that assessment of adult learners’ knowledge and experiences are necessary prior to additional education and training.

Therefore, staff developers and trainers could review the results of the checklists and have a working knowledge of audiences’ current level and depth of knowledge and experiences.

Training professionals could use the checklists when customizing a technical assistance program to meet specific program needs for local CACFP child care centers and FDCH providers. An ongoing use of these assessment resources could provide a system to follow up on training and/or technical assistance.

Although respondents expressed a slight concern over the time it took to complete the checklists, the checklists might serve as a useful guide to help a new child care provider acquire a quick view of industry expectations. State administrators and sponsors might encourage use of the checklists as one orientation tool or job aid along with other orientation materials provided to new hires.

The data in this study suggested that the FDCH respondents had a need for recognition. There was a 40% difference between FDCH and child care center respondents, when asked if they would be interested in receiving recognition from their sponsor or state agency for completing the checklists. The absence of demographic data for these groups does not permit a more detailed analysis of possible group differences. CACFP sponsors could further investigate procedures for professional recognition of outstanding achievement in nutrition services provided.

Child care professionals participating in the CACFP could benefit from a CPI process that is supported through training provided by state agencies and sponsors. The present study was conceived as an effort to develop a program self-assessment resource that would provide child care centers and FDCH providers with a solid first step in a CPI cycle. These results suggest that such an instrument could be of significant benefit to CACFP participants in evaluating current practices, program strengths and weaknesses, and as a tool for promoting program improvement. A particular advantage of the checklists is the relationship between item content and actual best practices of the industry. Additional studies may be helpful in refining the instruments and the application for improvement of foodservice delivery and services for child care centers and FDCHs.

Acknowledgments

This publication has been produced by the National Food Service Management Institute (NFSMI) – Applied Research Division, located at The University of Southern Mississippi with headquarters at The University of Mississippi. Funding for the Institute has been provided with Federal funds from the U.S. Department of Agriculture, Food and Nutrition Service, to The University of Mississippi. The contents of this publication do not necessarily reflect the views or policies of The University of Mississippi or the U.S. Department of Agriculture, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

The information provided in this manuscript is the result of independent research produced by NFSMI and is not necessarily in accordance with U.S. Department of Agriculture Food and Nutrition Service (FNS) policy. FNS is the federal agency responsible for all federal domestic child nutrition programs including the National School Lunch Program, the Child and Adult Care Food Program, and the Summer Food Service Program. Individuals are encouraged to contact their local child nutrition program sponsor and/or their Child Nutrition State Agency should there appear to be a conflict with the information contained herein, and any state or federal policy that governs the associated child nutrition program. For more information on the federal child nutrition programs please visit www.fns.usda.gov/cnd.