METHODOLOGY

What are Value-Added Methods?

The term “value-added” refers to the contributions teachers and schools make to student achievement. Value-added methods provide a way to measure this contribution.

Value-added methods are closely related to the statistical models traditionally used in formal quasi-experimental evaluation studies. However, value-added models differ from traditional models in several ways.

Value-added analyses are typically based on student and school information obtained from district and state administrative records and databases, as opposed to researcher-administered assessments, surveys, and interviews. As a result, information on student and family characteristics, attitudes, and activities may be limited. However, these databases typically include the universe of all students in a district or state rather than small samples, and they typically track students over many grades and years (including students who change schools).
Outcome measures such as standardized test scores may not be measured on a consistent scale across grades and over time. Yet students may pay greater attention to these assessments—administered as a part of ongoing accountability systems, where student performance matters—than to assessments administered solely for research purposes.

How is VARC’s Approach Unique?

For years education researchers and social scientists have sought a means to measure the contributions of teachers, administrators, and intervention programs. Rob Meyer, Director of the Value-Added Research Center, has developed methods that now make this possible. Several features make Meyer’s method unique:

It explicitly addresses student level selection bias and provides solutions for eliminating it.
It allows for possible decay over time in the effects of interventions.
It tracks student achievement across time.
It allows for the likelihood that school effects are no random, but may be correlated with school-level variables such as race, ethnicity, and income status. Such a correlation arises, for example, if poor students are disproportionally served by low-performing schools; this is an issue of policy interest.
It explicitly addresses measurement error in student achievement.

Once the value added by teachers, principals, intervention programs, and school reform efforts can be measured, it is possible to use this data to make decisions about staff professional development, about educator compensation programs that can increase student success, and about whether or not specific interventions are valuable. All these decisions can be used to initiate and support educators’ efforts at school reform.

Benefits of Value-Added Methods

Value-added methods “get the story right” by correcting for errors in the test scales, identifying and adjusting for bias in the administration of the test, in student participation, or in classroom treatments. In addition, one of the overriding goals of the work is to be transparent and fair. It is in everyone’s interest for schools to be as productive as possible for all students.

All teachers should be able to deeply understand and discuss the impact of changes in practice and curriculum for themselves and their students. Leaders should be able to make resource allocation decisions (money, staffing, etc.) informed by the best available data. Value-added methods can both showcase high levels of achievement as well as reward those who have mastered the art of improvement.

Current Areas of Activity

The Value-Added Research Center concentrates on the following areas of activity:

1. Development and implementation of the value-added model (currently at T3 - three years of test data and down to the classroom level). This includes production value-added models for Madison Metropolitan School District, Milwaukee Public Schools, and Chicago Public Schools. VARC is also developing a prototype model for Wisconsin’s Department of Public Instruction to demonstrate the use of value-added at the state level.

As part of their research, VARC scientists traditionally develops complex diagnostics and model refinements to test whether simplifying assumptions in the model are acceptable. They do this in-house and then share results with district staff, since the implications of violating model assumptions are really the district’s call to interpret. This level of engagement is what drives the research model, including addressing concerns about how best to handle mobile students and teachers, different testing windows, etc.

2. Data structures, data systems, and data quality. Many districts and states find that while their data systems might be adequate for annual compliance reporting, they are are not up to the task of real longitudinal assessment for students and adults. VARC works with SEA/LEA staff and vendors to examine the technical tools and human process used to collect, manage, analyze, and report out information. Often, the primary task is making clear how the requirements of longitudinal analysis test assumptions of data quality across all information systems. VARC shares solutions deployed in other settings and works with local teams do develop fixes for unique contexts.

3. Evaluation and intervention strategies. VARC has been working broadly in Milwaukee on a variety of evaluation projects to determine the effectiveness of many different programs. They are about to launch a new project that would require that senior leadership develop an "evaluation impact statement" for every major initiative. All interventions would be evaluated for effectiveness. They would also be deployed in ways that maximized the district's ability to judge effectiveness - including random assignment studies.

4. Professional development. VARC has developed a core Professional Development curriculum for a train-the-trainer model to build a PD cadre for the Milwaukee Public Schools. This PD both delivers training about the uses of assessment data and compares and contrasts the different uses for attainment and value-added outcome measures. VARC is very likely to begin doing similar work in Chicago.

5. Technical assistance. VARC provides technical assistance to recipients of Teacher Incentive Fund grants from the U.S. Department of Education. Thirty-four grantees across the U.S. receive a total of $95 million annually to support pay for performance systems. Along with Vanderbilt University, VARC provides technical assistance to the Department and individual grantees as they develop and deploy their programs. VARC provides assistance an all of the four areas outlined above. In addition, VARC has recruited consultants from other research areas to provide additional support for teacher and principal evaluation - a major component of pay for performance systems.