Most states have developed or revised school accountability systems in response to the Every Student Succeeds Act (ESSA). While these systems include multiple indicators, to many stakeholders the outcome of central interest is the overall rating or classification produced for each school. These ratings are often used to identify schools that merit reward or require support and to evaluate the efficacy of educational programs and policies.
States vary in their approach to producing school ratings. In some states, the accountability system culminates in a state-specific classification such as an A-F letter grade, awarding one to five stars, or other designations that communicate performance to the public. Other states do not provide an overall rating apart from the ESSA required categories of Targeted Support and Improvement (TSI), Additional Targeted Support and Improvement (ATSI), and Comprehensive Support and Improvement (CSI). Whether an overall or composite rating is provided, many states communicate performance using report cards or “dashboards” that often describe indicator level performance in terms thresholds (e.g., high/low; met expectations/ did not meet expectations; etc.).
Given the central importance of the accountability rating at the indicator or overall level, it is only reasonable to require compelling evidence that the rating has a high degree of validity for its intended interpretation and uses. A substantial part of that validity argument is in the design and implementation of a sound process for establishing performance standards that credibly reflects the state’s vision for the accountability system.
While there is a substantial research base in support of standard setting for assessments, very little attention has been given to establishing performance expectations in the context of school accountability systems. In many cases, accountability ratings are set normatively (e.g., the top 10% of schools receive an ‘A’), but using such procedures alone fails to ensure that the system reflects the policy values and prioritized outcomes that have been established by state leaders. In this context, this paper outlines recommended principles to guide the establishment of a standard setting-process for accountability systems and describes a framework for implementing standard setting.
The principles that guide an accountability standard-setting process are rooted in the Standards for Educational and Psychological Testing (2014) and include:
- Document rationale, procedures, and results;
- Ensure the process allows participants to apply their knowledge and experience; and,
- Include information associated with relevant consequences and criteria.
The framework is a systematic process to establish accountability performance standards that includes the following steps:
- Policy Definitions (PD). The state starts by deciding what performance categories should be established and a general Policy Definition for each category. For example, if the system will produce five performance levels (e.g., one to five stars, or letter grades A-F), then a brief description of each level consistent with the objectives of the system should be produced. These definitions should include any consequences associated with the level.
- School performance level descriptors (SPLDs). Next, the state should produce more specific School Performance Level Descriptors, which are detailed descriptions of what it looks like for a school to achieve each performance level in the state system. The SPLDs are based on the PDs but are written at a level of detail that can be used to inform the decision of panelists participating in the standard setting event. SPLDs should make clear whether performance across indicators, measures, and student groups is conjunctive, compensatory, or disjunctive to reflect statutory requirements and the intent of the system design. Additional guidelines for developing SPLDs are provided in the full paper.
- Standard Setting Panel. The state may then convene a broad-based panel of leaders and stakeholders to evaluate information and make recommendations regarding performance expectations for the accountability system. The goal is to assemble a panel that is broadly representative of the state’s interests and able to articulate a vision for education in the state.
- Standard Setting Preparation. In preparation for the standard setting event, the state should generate multiple documents and resources that are needed to implement the standard-setting process. These include meeting agendas, facilitator scripts, presentations, as well as various handouts given to panelists.
- Standard Setting Event. In convening the standard setting event, the state should identify a skilled and experienced facilitator who is very familiar with all aspects of the state system and context, has worked closely in developing the PDs and SPLDs, and can both operate and be perceived as independent. The event should include the following activities with the panelists:
- Review and elaborate SPLDs
- Independently identify threshold schools for each category
- Establish group recommendations
- Evaluate and document
Each of these activities is described in more detail in the full paper. Case studies from two states, Nevada and Utah, that implemented the described accountability standard-setting process, are also presented in the full paper to illustrate promising state practices.
In light of the importance of overall accountability school classifications, the process for establishing performance thresholds should be based on a well-defined, defensible procedure that reflects the state’s vision for the system. Doing so bolsters the validity of the system for supporting the intended interpretations and uses. This document may be helpful to education leaders seeking to develop a process for establishing accountability performance classifications that fit their system characteristics and policy priorities.