Overview (UC Davis)

The UCD formal assessments selected the theme of “Teacher Analysis”, and the following UFAC components under the above theme:

        1. Teacher examines student errors (e.g., a slip, misconception, misunderstanding) by forming
        and then revisiting their preliminary hypotheses as he/she gathers additional information (UFAC component #52).

        2. Teacher determines necessary adjustments to their instruction by analyzing evidence of student learning (measured progress) (UFAC component #77).


Participants

A Power analyses with 80% power and with the Type I error rate at .05 with MDES of 0.25 standard deviation yielded a sample size of 250 students for treatment and 250 for the control group. With an average of 25 students per class, we needed 20 Grade 8 classes, 10 classes for the treatment and 10 for the control groups. We contacted schools and discuss the possibility of testing 2 grade 8 classes per school. A total of 10 schools each with at least two grade 8 classes have been identified. Our plan is to design and field test the instrument and prepare training material for early fall, 2015 implementations of Component number 52 and 77.


School demographics

We targeted schools with large number of ELLs and high percent of students receiving free/reduced lunch program. These schools that are considered at risk provide opportunity to observe the potentials of formative assessments.


Procedures

Both treatment and control students in the sampled school will be administered the mathematics pre-test and post-tests that are constructed for this study based on Common Core State Standards, more specifically 8EE5, 8EE6, 8EE7a and 8EE7b. Results of pre-test are used for two purposes in the UC Davis UFAC implementation. 

First, the pretest scores will be use as covariates to control for possible initial differences between the treatment and the control groups and second, they provide formative information for the teachers in the treatment, i.e., inform teachers of areas that students need more attention. 

The information teachers receive from the pre-test as formative assessment helps them understand their individual student’s needs for the particular subject areas. Teachers are trained to carefully review performance of each student and make necessary adjustments to their instruction. With this initial adjustment, teachers continue for a week or two. 

They will then receive information on students’ performance based on the first interim assessment. Based on the results of the first interim assessments, teachers make further adjustments to their instruction. Teachers will then continue with the adjusted instruction for another week or to and will then receive the second interim assessment. Adjustments will be made based on multiple formative assessments. 

At the end of instructions for these units, a summative assessment will be given to assess students’ progress on the units (8EE5, 8EE6 and 8EE7). The control group, on the other hand, takes the pre-test and the posttest without sharing the results with the teachers. The effectiveness of the selected UFAC Components will then be determined by comparing the summative assessment results across the treatment and the control groups.

During the implementation phase of the UFAC Components, a survey of “Fidelity of Implementation” will be given to teachers multiple times, one before receiving the first interim assessment results, one before receiving the second interim assessment results and one at the end of the units before the summative assessments. Results of these surveys along with results from other surveys will be used a conditioning variables in the comparisons between the treatment and control groups.


Teacher Recruitment

Teachers for both treatment and control groups were identified through districts. Districts were asked to provide access to grade 8 classes in the district that have a large number of students at risk (such as ELLs, students with learning disabilities and students from low income families). Twenty grade 8 classes have been (and continue to be) identified for the treatment and control groups. We identified four additional classrooms in case of attrition, i.e., some of the classes withdraw from the study. Since teachers in the treatment group have to be extensively involved in this experiment, we offered each teacher $500 to compensate for their time and efforts. Similarly, we offered $300 to each control group classes for their participation in the study.

Training and Support. Teachers in the treatment group have to read the instructions for this experiment very carefully and act accordingly. Therefore, detailed trainings will be provided to the treatment group teachers. They will be instructed how to review and interpret the results of pretest, two interim tests and posttest and how to adjust their instruction based on the feedback they receive from the test outcomes.


Instruments

Pretests. The pretest items were designed to assess prior grade common core standards such as fifth grade graph reading (5G2), sixth and seventh grade rational number operation and number sense (CC6NS2, 7NS1), and sixth and seventh grade expressions and equations knowledge (6EE9, 7EE3). Multiple assessment sources were reviewed including National Assessment of Educational Progress (NAEP), California and Connecticut State Departments of Education, mathematics education research (Alexander & Ambrose, 2010; Carpenter, et al., 1999) and the researchers’ past teaching experience. Sample pretest items with corresponding prerequisite standards and item sources are shown in the accompanying attachment (accomplishments, attachment 2).

Interim tests. The interim tests were designed to assess students’ developing skills as they progress through standards 8EE5, 6 and 7. The interim tests were designed to be utilized by the UCD treatment classrooms only, and not used by the UCLA treatment schools. Sample interim test items with corresponding standards and item sources are shown in the accompanying attachment.

Posttests. The summative items were designed to assess mastery of the three standards, 8EE5, 8EE6 and 8EE7a&b. For the UCLA treatment classrooms, the posttest was adjusted to assess 8EE 5 and 8EE6 only. Sample posttest items with corresponding covered standards and item sources are shown in the accompanying attachment.

Rubric. In the UCD treatment design, teachers were supplied with a rubric to score the formal assessments. This rubric combined the correct solution, the rubric point value of each question and further tools for teachers. These tools included an error analysis column which listed multiple possible misconceptions that may have led to particular errors in students’ solutions. Additionally, teachers were supplied with next step suggestions for reteaching and rethinking. These next steps included conceptual and skill-based ideas. For example, teachers were given questions to explore conceptual understanding, such as: “What makes relationships proportional?” Interpretation skills were addressed with questions such as: “How are these equations the same and different? Do they act differently?” And calculation skills were suggested such as: “Define proportion and calculate slope. Identify positive and negative slopes.”

The pretest, interim test and posttest rubrics also included descriptions of required and optional explanations such as: “A sufficient explanation includes a statement of fact such as: ‘Both lines are not proportional’ or gives a reason such as: ‘Because the graph does not go through the origin.’” Also optional explanations and calculation methods were shown. These error analysis and next step tools were designed to assist treatment teachers to utilize students’ answers to formal assessments to use formal assessments for formative assessment purposes. The error analysis is supported by mathematics education research experts (Alexander, 2013). Sample items from the pre test rubric with error analysis and next steps are shown in the accompanying attachment.


Design of Analyses

The main dependent variable in this study is the mathematics posttest and the main independent variable is the treatment group membership (treatment vs. control). Therefore, the principal analysis would be comparing the performance of treatment and control groups in the mathematics posttest. However, there are many other variables that may explain the performance difference between the treatment and control groups. First, in spite of random assignment of subjects to treatment and control groups some initial differences may remain. Therefore, some of the variables such as students’ pretest, their prior year state test scores can be used as covariates. Due to multivariate nature of this study and due to the fact that there are multivariate covariate and multivariate dependent variables, the analyses will be based on latent variables modeling rather than single structure variables. The use of latent variable helps to have a more reliable and valid measures by focusing on the common variance between groups of variables that are highly correlated.

As mentioned earlier in this report, different instruments (such questionnaires for students’ background variables, fidelity of implementation of treatment, etc.) are used in this study. These data will be used as conditioning variables in a series of confirmatory factor analyses. Also, a pretest/posttest comparison is not used due to the content differences. The pretest assesses prerequisite standards knowledge and the posttest assesses summative content knowledge of the 8EE units of instruction. For example, the pretests may indicate student aptitudes and could be applied to both UCD (formal test treatment) and UCLA (task treatment) students. By design, the pretest is a rich source of formative assessment information for teachers in UCD treatment classrooms.