Designing New Tools to Measure Student Learning in Schools

DC PCSB is developing a new accountability framework that will focus on the learning outcomes for all DC students, especially those from historically marginalized groups. In our new Strategic Roadmap that will guide our work in the coming years, we are committed to focusing on equity. To this end, our Strategic Roadmap focuses on three priorities: Excellent School, Enduring Partnerships, and Effective Organization. One of the Excellent Schools priorities includes piloting and implementing a new accountability framework.  

In this blog series, we will discuss our efforts to engage families, school leaders, and partners and share how we will measure school quality, progress, and environment.  

***

Engaging School Leaders 

DC PCSB continues to explore new ways to engage school leaders. This begins with prioritizing the voices of school leaders that educate students in historically marginalized groups. To do this, DC PCSB conducted multiple small engagement sessions with a handful of public charter school leaders. These sessions allowed leaders to strengthen relationships, be heard, speak candidly about the barriers and challenges of student outcomes. 

These sessions paved the way for creating two advisory groups, one PK-12 group, and an Adult Education group. Each group was limited to one representative from each public charter school network. This approach ensures that all ideas, opinions, and beliefs get heard. Before the meeting, a list of guiding questions was shared with participants ahead of time to allow for a rich discussion and encourage full participation. 

To date, more than 25 small group meetings have been held and a survey was disseminated.  

Here is a timeline of activities related to the development: 

We will share more about the activities in future blogs. 

What School Leaders Had to Say: PK-12 + Adult PCS Discuss Accountability

In the spring of 2021, DC PCSB held monthly small engagement sessions to discuss high-level conceptual changes and priorities to the accountability framework. Each session included up to one representative per LEA. The accountability sessions are an opportunity to receive feedback about potential changes to the current Performance Management Framework (PMF), read more about it here. 

The information gathered from these engagement sessions assisted us in the curation of our next phase of engagement, which will focus on the technical aspect of collecting and reporting these potential data points. 

These sessions are part of an ongoing engagement process that will continue as we draft the new pilot accountability framework guidance. 

This summary represents the high-level takeaways heard from all the sessions:

Key takeaways from PK-12 PCS Sessions:

• Public Charter LEAs have concerns around the implications of altering the purpose of the assessments from an internal tool used to inform instruction to one that informs DC PCSB’s accountability decisions, adding new measures to identify growth, and increasing testing in order to collect measures
• Limiting a public charter school’s flexibility by using standardized test results
• Public Charter LEAs would like to find new approaches to understand growth in student populations 

Key takeaways from Adult PCS Sessions:

• Public Charter LEAs discussed concerns with reaching students (e.g., students’ phone numbers constantly changing)
• Attendance metrics fail to capture the engagement of the adult learner due to the structure of course offerings for various adult programs; adult LEAs focus on outcomes not in-seat attendance
• Finding ways to better capture synchronous and asynchronous learning
• Bringing more awareness to completion of certifications that are not just industry certifications, emphasizing the importance of the micro credentials necessary for the outcome (e.g., state-level certifications like cosmetology, barber, etc)

Based on the takeaways from PK-12 and Adult PCS leaders, DC PCSB will include the information in the pilot accountability framework:

• Provide an in-depth explanation of each growth measure and what it will achieve

• Celebrate the milestones of adult students in the adult education framework 

• Revise the measurement for the attendance indicators 
• Include school selected nationally normed growth assessments in the business rules

Check back here for monthly updates.  

Descriptions of Metrics Collected for Model Performance Framework

This Data Collection Summary is an inventory description of the data that will be collected for modeling potential approaches to the new framework. Not every data point will be used in the Revised Accountability Framework. Data analyses and business rule modeling will inform discussions with stakeholders to make decisions on floors, targets, and weights for the new framework. 

The Revised Accountability Framework will be detailed in a new technical guide, slated for release in Spring 2023 and will be fully implemented in school year 2023-24.

  Changes in Measures

DC PCSB is expanding existing measures and exploring alternative ways of capturing existing data. 

*These already exist in the PMF as display only measures.

The only measure we have removed is attendance for adult learners. Adult learners are non-compulsory and are not required by law to attend. We will continue to capture adult learner persistence and outcomes.

Inventory Description of Data Points Collected While Modeling Approaches for New Framework

Median Growth Percentile (MGP) and School Selected Nationally Normed Growth Assessments

Historically, DC PCSB has used the MGP on the PARCC as the growth measure which assesses the relative year-to-year progress made by individual students at a school. School selected nationally normed growth assessments include assessments such as i-Ready or NWEA MAP where LEAs will be able to choose between these nationally normed options for this growth measure during the pilot and DC PCSB will explore both options and share with LEAs the findings of each to inform conversations around growth measures for accountability. 

On Track Measures

Historically, DC PCSB has collected data for on track measures for ninth grade students. This measure aims to identify the percentage of ninth grade students earning enough credits to be on track for high school graduation within four years. DC PCSB is exploring additional on track measures for grades 10 and 11 to better understand the relationship between these measures and on time graduation rates.

 College and Career Acceptance

Traditionally, DC PCSB has collected college acceptance data from LEAs. Expanding this measure to capture career post-secondary pathways allows students to be recognized for several successful pathways available to them. Career pathway guidance is still being developed; however, this is intended to include acceptance into the military, the police academy, fire academy, etc. This would likely not be a new measure, but become part of the college acceptance measure, with students either having a college or career program acceptance.

Adult Learner Certifications

Adult LEAs provide certification pathways including Carpentry Level 1, National Construction Career Test, Child Development Associate (CDA), Certified Nursing Assistant (CAN), Leadership in Energy and Environmental Design (LEED), NAFTrack, NAFTrack Certified Hiring. DC PCSB is exploring criteria for additional certifications to be included in this measure and will share guidance with LEAs once these criteria are developed.

 Attendance

Historically, DC PCSB has captured in-seat attendance, which captures the percentage of school days students are “in their seats” across the entire school year.

 DC PCSB is exploring chronic absenteeism as a possible alternative attendance measure. Chronic absenteeism captures the number of students who are absent (excused or unexcused) for 10 percent or more of the school's days on which the student was enrolled across the entire school year.

The pilot will explore this data alongside LEAs to identify which attendance measure best captures attendance for public charter schools.

Display Only

Display only measures are not included in the overall calculation of an accountability score. These measures, where available, may be included in the framework to provide additional context for community members to make decisions on where they’d like to attend or enroll their student.

 DC PCSB also continues to explore the following measures: discipline by student group, teacher tenure and demographics, Qualitative Site Reviews, and mission-specific goals. Early childhood performance is already displayed in the accountability framework and includes data around student performance using the LEAs chosen early childhood assessment.