Evaluation Assessment Tools:
Measuring Quality Afterschool Programming

Source: Measuring Youth Program Quality: A Guide to Assessment Tools, Nicole Yohalem and Alicia Wilson-Ahlstrom, The Forum for Youth Investment, with Sean Fischer and Marybeth Shinn, New York University, March 2007

Assessment of Afterschool Program Practices Tool
Developed by NIOST and the Massachusetts Department of Education

Overview: The Assessment of Afterschool Program Practices Tool (APT) is designed to help practitioners examine and improve what they do in their program to support young people’s learning and development. It examines those program practices that research suggests relate to youth outcomes (e.g., behavior, initiative, social relationships). A research version of the APT (the APT-R) was developed in 2003-2004. This more user-friendly self-assessment version was developed in 2005.
Primary Purpose(s): Program Improvement; Monitoring/Accreditation
Program Target Age: Grades K – 8
Relevant Settings: Both structured and unstructured programs that serve elementary and middle school students during the non-school hours.
Content: The APT measures a set of 15 program-level features and practices that can be summarized into five broad categories – program climate, relationships, approaches and programming, partnerships and youth participation.

Out-of-School Time Program Observation Tool
Developed by Policy Studies Associates, Inc.

Overview: The Out-of-School time Program Observation Tool (OST) was developed in conjunction with several research projects related to out-of-school time programming, with the goal of collecting consistent and objective data about the quality of activities through observation. Its design is based on several assumptions about high-quality programs – first that certain structural and institutional features support the implementation of high-quality programs and second that instructional activities with certain characteristics – varied content, mastery-oriented instruction and positive relationships – promote positive youth outcomes.
Primary Purpose: Research/Evaluation
Program Target Age: Grades K – 12
Relevant Settings: Varied school- and community-based after-school programs.
Content: The OST documents and rates the quality of the following major components of after-school activities: interactions between youth and adults and among youth, staff teaching processes, and activity content and structures.

Program Observation Tool
Developed by the National AfterSchool Association

Overview: The Program Observation Tool is the centerpiece of the National AfterSchool Association’s (NAA) program improvement and accreditation process and is designed specifically to help programs assess progress against the Standards for Quality School-Age Care. Developed in 1991 by NAA and the National Institute on Out-of-School Time, the tool was revised and piloted before the accreditation system began in 1998.
Primary Purpose(s): Program Improvement; Monitoring/Accreditation
Program Target Age: Grades K – 8
Relevant Settings: School and centerbased Afterschool programs.
Content: The Program Observation Tool measures 36 “keys of quality,” organized into six categories. Five are assessed primarily through observation: human relationships; indoor environment; outdoor Environment; activities; and safety, health and nutrition. The sixth – administration – is assessed through questionnaire/ interview. The tool reflects NAA’s commitment to holistic child development and its accreditation orientation.

Overview: The Program Quality Observation (PQO) was designed to help observers characterize the overall quality of an after-school program environment and to document individual children’s experiences within programs. The PQO has been used in a series of research studies and has its roots in Vandell’s observational work in early child care settings.
Primary Purpose: Research/Evaluation
Program Target Age: Grades 1-5
Relevant Settings: Varied school- and community-based after-school programs.
Content: The PQO focuses primarily on social processes and in particular, three components of quality of children’s experiences inside programs: relationships with staff, relationships with peers and opportunities for engagement in activities.

Program Quality Self-Assessment
Developed by the New York State Afterschool Network

Overview: The Program Quality Self-Assessment Tool (QSA) was developed exclusively for self-assessment purposes (use for external assessment and formal evaluation purposes is discouraged). The QSA is intended to be used as the focal point of a collective self-assessment process that involves all program staff. Soon after it was created in 2005, the state of New York began requiring that all 21st CCLC-funded programs use it twice a year for self-assessment purposes.
Primary Purpose: Program Improvement
Program Target Age: Grades K – 12
Relevant Settings: The full range of school and community-based after-school programs. The QSA is particularly relevant for programs that intend to provide a broad range of services as opposed to those with either a very narrow focus or no particular focus (e.g., drop-in centers).
Content: The QSA is organized into 10 essential elements of effective Afterschool programs, including environment/climate; administration/organization; programming/activities; and youth participation/ engagement, among others. A list of standards describes each element in greater detail. The elements represent a mix of activity-level, program-level and organizational-level concerns.

Promising Practices Rating Scale
Developed by the Wisconsin Center for Education Research and Policy Studies Associates, Inc.

Overview: The Promising Practices Rating Scale (PPRS) was developed in the context of a study of the relationship between participation in high quality after-school programs and child and youth outcomes. The tool was designed to help researchers document type of activity, extent to which promising practices are implemented within activities and overall program quality. The PPRS builds directly on earlier work by Deborah Lowe Vandell and draws upon several other observation instruments included in this report.
Primary Purpose: Research/Evaluation
Program Target Age: Grades K – 8
Relevant Settings: Varied school- and community-based after-school programs.
Content: The PPRS focuses primarily on social processes occurring at the program level (other tools in the PP assessment system are available to collect other kinds of information). The tool addresses activity type, implementation of promising practices, and overall program quality. The practices at the core of the instrument include supportive relations with adults, supportive relations with peers, level of engagement, opportunities for cognitive growth, appropriate structure, over-control, chaos and mastery orientation.

Quality Assurance System™
Developed by Foundations Inc.

Overview: The Quality Assurance System™ (QAS) was developed to help programs conduct quality assessment and continuous improvement planning. Based on seven “building blocks” that are considered relevant for any Afterschool program, this Web-based tool is expandable and has been customized for particular organizations based on their particular focus. The QAS focuses on quality at the “site” level and addresses a range of aspects of quality from interactions to program policies and leadership.
Primary Purpose: Program Improvement
Program Target Age: Grades K – 12
Relevant Settings: A range of school and community-based programs.
Content: The various components of quality that the QAS focuses on are considered “building blocks.” The seven core building blocks include: program planning and improvement; leadership; facility and program space; health and safety; staffing; family and community connections; and social climate. Three additional “program focus building blocks” that reflect particular foci within programs are also available.

School-Age Care Environment Rating Scale
Developed by FPG Child Development Institute, UNC & Concordia University, Montreal

Overview: The School-Age Care Environment Rating Scale (SACERS), published in 1996 and updated periodically, is one of a series of quality rating scales developed by researchers at the Frank Porter Graham Child Development Institute. SACERS focuses on “process quality” or social interactions within the setting, as well as features related to space, schedule and materials that support those interactions. The SACERS can be used by program staff as well as trained external observers or researchers.
Primary Purpose(s): Program Improvement; Monitoring/Accreditation; Research/Evaluation
Program Target Age: Grades K – 8
Relevant Settings: A range of program environments including child care centers, school-based after-school programs and community-based organizations.
Content: SACERS is based on the notion that quality programs address three “basic needs”: protection of health and safety, positive relationships, and opportunities for stimulation and learning. The seven sub-scales of the instrument include space and furnishings; health and safety; activities; interactions; program structure; staff development; and a special needs supplement.

Youth Program Quality Assessment
Developed by the High/Scope Educational Research Foundation

Overview: The overall purpose of the Youth Program Quality Assessment (YPQA) is to encourage individuals, programs and systems to focus on the quality of the experiences young people have in programs and the corresponding training needs of staff. While some structural and organizational management issues are included in the instrument, the YPQA is primarily focused on what the developers refer to as the “point of service” – the delivery of key developmental experiences and young people’s access to those experiences.
Primary Purpose(s): Program Improvement; Monitoring/Accreditation; Research/Evaluation
Program Target Age: Grades 4 – 12
Relevant Settings: Structured programs in a range of school- and community-based settings.
Content: Because of the focus on the “point of service,” the YPQA emphasizes social processes – or interactions between people within the program. The majority of items are aimed at helping users observe and assess interactions between and among youth and adults, the extent to which young people are engaged in the program, and the nature of that engagement. However the YPQA also addresses program resources (human, material) and the organization or arrangement of those resources within the program.

Teaching Social Skills
Terese Dana, Behavioral specialist and social skills instructor

Brief Resource Overview:  This article defines social skills, gives reasons to teach them, and lists 14 tasks specific to children ages 6-12.

Quality Standards Cited:  Supports NAA standards:

1. Staff encourage children and youth to make choices and to become more responsible;
2. Staff interact with children and youth to help them learn;
3. Children and youth generally interact with one another in positive ways.

Quality Indicators Cited: 
Staff encourage children and youth to make choices and to become more responsible

1. Staff offer assistance in a way that supports a child’s initiative
2. Staff assist children without taking control, and they encourage children to take leadership roles
3. Staff give children many chances to choose what they will do, how they will do it, and with whom
4. Staff help children make informed and responsible choices

Staff interact with children and youth to help them learn

1. Staff ask questions that encourage children to think for themselves
2. Staff share skills and resources to help children gain information and solve problems
3. Staff vary the approaches they use to help children learn
4. Staff help children use language skills through frequent conversations

Children and youth generally interact with one another in positive ways

1. Children appear relaxed and involved with each other
2. Children show respect for each other
3. Children usually cooperate and work well together
4. When problems occur, children often try to discuss their differences and work out a solution

Empirical Support for Family Involvement in Education: Success for
All Students
Children, Youth, and Family Consortium

Brief Resource Overview:  This article is an overview on family-school collaboration in order to support student outcomes.  It also gives suggestions for enhancing program development.

Quality Standards Cited:  Supports NAA standards
Staff, families, and schools share important information to support the well-being of children and youth

Quality Indicators Cited:
Staff, families, and schools share important information to support the well-being of children and youth

1. Program policies require that staff and family members communicate about the child’s well-being
2. Staff, families, and schools work together as a team to set goals for each child; they work with outside specialists when necessary
3. Staff and families share information about how to support children’s development
4. Staff and families join together to communicate and work with the schools 

Dramatic Play in School Age Care Environments
Duane Seibel, Human Services Program Instructor at Selkirk College in Castlegar, British Columbia

Brief Resource Overview:  This article discusses how children learn social norms (both positive and negative) through their experiences, play, media, etc.

Quality Standards Cited:  Supports NAA standards

1. Activities reflect the mission of the program and promote the development of all the children and youth in the program
2. There are sufficient materials to support program activities.

Quality Indicators Cited:
Activities reflect the mission of the program and promote the development of all the children and youth in the program

1. Activities are in line with the styles, abilities, and interests of the individuals in the program
2. Activities are well suited to the age range of children in the program
3. Activities reflect the languages and cultures of the families served
4. Activities reflect and support the program’s mission

There are sufficient materials to support program activities

1. Materials are complete and in good repair
2. There are enough materials for the number of children in the program
3. Materials are developmentally appropriate for the age range of the children in the program
4. Materials promote the program’s mission

Preventing Mental Disorders in School Age Children: A Review of the Effectiveness of Prevention Programs
Mark T. Greenberg, Ph.D.; Celine Domitrovich, Ph.D.; and Brian Bumbarger

Brief Resource Overview:  This article highlights that preventive intervention programs have been successful with school-aged youth in reducing psychological symptoms and/or related behaviors.  The article also identifies characteristics shared by effective preventive intervention programs.

Quality Standards Cited:  Supports NAA standards

1. Staff, families, and schools share important information to support the well-being of children and youth
2. The program builds links to the community.

Quality Indicators Cited:
Staff, families, and schools share important information to support the well-being of children and youth

1. Program policies require that staff and family members communicate about the child’s well-being
2. Staff, families, and schools work together as a team to set goals for each child; they work with outside specialists when necessary
3. Staff and families share information about how to support children’s development
4. Staff and families join together to communicate and work with the schools

The program builds links to the community

1. Staff provide information about community resources to meet the needs of children and their families
2. The program develops a list of community resources. The staff draw from these resources to expand program offerings
3. The staff plan activities to help children get to know the larger community
4. The program offers community-service options, especially for older children

Summary of Major Youth Serving Programs Evaluated Using Experimental
and Quasi-Experimental Methods
Jacquelynne S. Eccles and Janice Templeton

Brief Resource Overview:  This article is a summary of programs for youth ages 10-18 which indicates key program characteristics as well as social characteristics that facilitate positive youth development. 

Quality Standards Cited:  Supports NAA standards

1. Staff interact with children and youth to help them learn
2. Staff use positive techniques to guide the behavior of children and youth
3. Activities reflect the mission of the program and promote the development of all the children and youth in the program.

Quality Indicators Cited:
Staff interact with children and youth to help them learn

1. Staff ask questions that encourage children to think for themselves
2. Staff share skills and resources to help children gain information and solve problems
3. Staff vary the approaches they use to help children learn
4. Staff help children use language skills through frequent conversations

Staff use positive techniques to guide the behavior of children and youth

1. Staff give attention to children when they cooperate, share, care for materials, or join in activities
2. Staff set appropriate limits for children
3. Staff use no harsh discipline methods
4. Staff encourage children to resolve their own conflicts. Staff step in only if needed to discuss the issues and work out a solution

Activities reflect the mission of the program and promote the development of all the children and youth in the program

• Activities are in line with the styles, abilities, and interests of the individuals in the program
• Activities are well suited to the age range of children in the program
• Activities reflect the languages and cultures of the families served
• Activities reflect and support the program’s mission.

Afterschool Training Toolkit: Building Quality Enrichment Activities
National Partnership for Quality Afterschool Learning 

Brief Resource Overview: 
This toolkit is designed to provide afterschool staff and partners the resources they need to develop fun, innovative, and academic enrichment activities that engage students and extend students’ knowledge in new ways and that increases students’ academic achievement.  The toolkit includes standards-based multi-media resources such as research-based practices, sample lessons, interactive activities, and afterschool program video segments.  Content areas addressed within the toolkit include arts, literacy, match, science, technology, and homework assistance.

Quality Standards Cited: 

1. Effective Programming
2. Program Planning and Structure
3. Assessment
4. Evaluation and Accountability

Concept to Classroom: Afterschool Programs – From Vision to Reality
Thirteen Ed Online

Brief Resource Overview: 
This resource is an online workshop.  The workshop highlights the benefits that a quality afterschool program can provide to students, their families, and their community, and discusses ways to create and sustain quality afterschool programs.  The workshop is split into four sections: explanation, demonstration, exploration, and implementation.

Quality Standards Cited: 

1. Set goals that reflect the needs of the children and families in your community.
2. Establish strong leadership and a clear organizational structure.
3. Develop a budget and financial plan that provides for long-term sustainability.
4. Understand the legal requirements and liability issues for your afterschool programs, and be sure your plan addresses them.
5. Address the safety, health, and nutritional issues that face the children in your program.
6. Carefully consider staffing needs. (Hiring and retaining skilled, qualified, caring staff people. Providing for professional development. Ensuring low adult/child ratios and small group sizes. Enlisting school day teachers as afterschool staff.)

Gaining a Voice After School: Why After-School Programs Are a Powerful Resource for English-Language Learners
Claudia Weisburd

Brief Resource Overview: 
This article highlights after-school programs as a key partner of schools in offering different language-learning opportunities that complement ELL teaching and learning during the school day.  After-school environments are communicative social settings that assist youth in broadening their language skills, developing as students, and connecting with schooling.

Quality Standards Cited: 

1. Extensive and Varied Opportunities to Practice
2. Effective Programming Designed for Different Learning Styles and Types of Participation
3. Supportive Adult and Peer Relationships
4. Family, School, and Community Involvement
5. Combined Professional Development for Non-ELL Teachers and After-School Staff

Evaluating After-School Care Research Brief
RAND Researchers Megan Beckett, Angela Hawken, and Alison Jacknowitz

Brief Resource Overview: 
This research brief highlights model after-school program characteristics or practices that are supported by scientific studies or by professional judgment.

Quality Standards Cited: 

1. Staff Management Practices
2. Program Management Practices
3. Communication with Other Organization

Quality Indicators Cited: 
Staff Management Practices
1) Hiring and retaining educated staff
2) Providing attractive compensation
3) Training staff

Program Management Practices
1) Ensuring that programming is flexible
2) Establishing and maintaining a favorable emotional climate
3) Establishing clear goals and evaluating programs accordingly
4) Keeping total enrollment low
5) Maintaining a low child-to-staff ratio
6) Maintaining continuity and complementarity with regular day school
7) Paying adequate attention to safety and health
8) Providing a sufficient variety of activities
9) Providing adequate space
10)Providing age-appropriate activities and materials
11) Providing enough quality materials
Communications with Other Organizations
1) Involving families
2) Using community-based organizations and facilities
3) Using volunteers

Exemplary Practices in Afterschool Program Development:  Rubrics for Tracking Internal Progress
Center for Collaborative Solutions and the Community Network for Youth Development

Brief Resource Overview: 
The assessment tool identifies fourteen practices/standards of quality after-school programs.  Each of the fourteen practices/standards is accompanied by exemplary practice indicators.  This is an excellent tool for after-school programs to track and measure program quality.

Quality Standards Cited: 

1. A Vision that Clarifies Purpose and Ignites Passion
2. Integrating Leadership, Mentorship, Coaching and Management
3. Supporting Ongoing Staff Development and Training
4. Program Activities are Directly Linked to Youth Development Outcomes
5. Achieving Positive Academic Results
6. Diversity, Access, Inclusion and Equity
7. Promoting a Sense of Physical and Emotional Safety
8. Building Supportive Relationships
9. Creating Challenging and Engaging Learning Experiences
10. Meaningful Participation of Young People
11. Neighborhood and Community Connections
12. Consistently High Student Attendance
13. Measuring and Managing Outcomes
14. Secured Ongoing, Balanced and Diversified Funding Resulting in Financial Sustainability

Quality Indicators Cited: 
Please see document for a list of indicators corresponding to each standard.

The NAA Standards for Quality School-Age Care: Standards at a Glance
National AfterSchool Association

Brief Resource Overview: 
The brochure describes the practices that lead to stimulating, safe, and supportive programs for young people ages 5 to 14 during out-of-school time.  The brochure provides an overview of 6 standards categories, including the 2 standard categories of indoor and outdoor environments, which are organized into 36 standards to quality.  The first 20 standards are things that after-school staff can see happening in a program.  The additional 16 administration related standards describe the policies and practices that are the basis of a quality program.

Quality Standards Cited:
Indoor Environment

1. The program’s indoor space meets the needs of children and youth
2. The indoor space allows children and youth to take initiative and explore their interests

Outdoor Environment
        1)     The outdoor play area meets the needs of children and youth, and the equipment allows them to be independent and creative

Quality Indicators Cited: 
Indoor Environment
The program’s indoor space meets the needs of children and youth

1. There is enough room for all program activities
2. The space is arranged well for a range of activities: physical games and sports, creative arts, dramatic play, quiet games, enrichment offerings, eating, and socializing
3. The space is arranged so that various activities can go on at the same time without much disruption
4. There is adequate and convenient storage space for equipment, materials, and personal possessions of children and staff

The indoor space allows children and youth to take initiative and explore their interests

1. Children can get materials out and put them away by themselves with ease
2. Children can arrange materials and equipment to suite their activities
3. The indoor space reflects the work and interests of the children
4. Some areas have soft, comfortable furniture on which children can relax

Outdoor Environment
The outdoor play area meets the needs of children and youth, and the equipment allows them to be independent and creative

1. Each child has a chance to play outdoors for at least 30 minutes out of every three-hour block of time at the program
2. Children can use a variety of outdoor equipment and games for both active and quiet play
3. Permanent playground equipment is suitable for the sizes and abilities of all children
4. The outdoor space is suitable for a wide variety of activities

Quality Time After School: What Instructors Can do to Enhance Learning
Jean Grossman, Margo Campbell, and Becca Raley

Brief Resource Overview: 
This study builds on current knowledge about what constitutes engaging after-school programs in which youth of all ages learn and develop. The study details key enrichment characteristics linked to youth engagement and learning and provides after-school staff with a framework for how to improve program quality and develop engaging and effective learning environments for young people.

Quality Standards Cited: 

1. Staff effectively manage groups in ways that ensure youth feel respected by both the adults and other youth
2. Staff positively support youth and their learning process

Quality Indicators Cited: 
Staff effectively manage groups in ways that ensure youth feel respected by both adults and other youth

1. Youth report getting more out of an activity at each step in the learning process
2. Youth report they like the activity
3. Youth report they are more engaged in the day’s tasks
4. Youth report they learn more than youth in less well-managed activities

Staff positively support youth and their learning process

1. Youth report they enjoyed the activity
2. Youth report they feel more engaged and perceive they learned more than those who experienced less adult support

School-Age Care Environment Rating Scale
Thelma Harms,  Ellen Vineberg Jacobs, and Donna Romano White

Brief Resource Overview: 
Designed to assess group-care programs for children of school-age; ages 5 to 12.  The scale consists of 49 items, including 6 supplementary items for programs enrolling children with special needs.  The items are organized into the following 7 subscales: Space and Furnishings, Health and Safety, Activities, Interactions, Program Structure, Staff Development, and Special Needs.

Quality Standards Cited:
The scale identifies seven standard categories of space and furnishings, health and safety, activities, interactions, program structure, staff development, and special needs.
Please see scale for a list of indicators corresponding to each standard category.