The UCD formal assessments selected the theme of “Teacher Analysis”, and the following UFAC components under the above theme: 1. Teacher examines student errors (e.g., a slip, misconception, misunderstanding) by forming 2. Teacher determines necessary adjustments to their instruction by analyzing evidence of student learning (measured progress) (UFAC component #77).
A Power analyses with 80% power and with the Type I error rate at .05 with MDES of 0.25 standard deviation yielded a sample size of 250 students for treatment and 250 for the control group. With an average of 25 students per class, we needed 20 Grade 8 classes, 10 classes for the treatment and 10 for the control groups. We contacted schools and discuss the possibility of testing 2 grade 8 classes per school. A total of 10 schools each with at least two grade 8 classes have been identified. Our plan is to design and field test the instrument and prepare training material for early fall, 2015 implementations of Component number 52 and 77.
We targeted schools with large number of ELLs and high percent of students receiving free/reduced lunch program. These schools that are considered at risk provide opportunity to observe the potentials of formative assessments.
Both treatment and control students in the sampled school will be administered the mathematics pre-test and post-tests that are constructed for this study based on Common Core State Standards, more specifically 8EE5, 8EE6, 8EE7a and 8EE7b. Results of pre-test are used for two purposes in the UC Davis UFAC implementation. First, the pretest scores will be use as covariates to control for possible initial differences between the treatment and the control groups and second, they provide formative information for the teachers in the treatment, i.e., inform teachers of areas that students need more attention. The information teachers receive from the pre-test as formative assessment helps them understand their individual student’s needs for the particular subject areas. Teachers are trained to carefully review performance of each student and make necessary adjustments to their instruction. With this initial adjustment, teachers continue for a week or two. They will then receive information on students’ performance based on the first interim assessment. Based on the results of the first interim assessments, teachers make further adjustments to their instruction. Teachers will then continue with the adjusted instruction for another week or to and will then receive the second interim assessment. Adjustments will be made based on multiple formative assessments. At the end of instructions for these units, a summative assessment will be given to assess students’ progress on the units (8EE5, 8EE6 and 8EE7). The control group, on the other hand, takes the pre-test and the posttest without sharing the results with the teachers. The effectiveness of the selected UFAC Components will then be determined by comparing the summative assessment results across the treatment and the control groups. During the implementation phase of the UFAC Components, a survey of “Fidelity of Implementation” will be given to teachers multiple times, one before receiving the first interim assessment results, one before receiving the second interim assessment results and one at the end of the units before the summative assessments. Results of these surveys along with results from other surveys will be used a conditioning variables in the comparisons between the treatment and control groups.
Teachers for both treatment and control groups were identified through districts. Districts were asked to provide access to grade 8 classes in the district that have a large number of students at risk (such as ELLs, students with learning disabilities and students from low income families). Twenty grade 8 classes have been (and continue to be) identified for the treatment and control groups. We identified four additional classrooms in case of attrition, i.e., some of the classes withdraw from the study. Since teachers in the treatment group have to be extensively involved in this experiment, we offered each teacher $500 to compensate for their time and efforts. Similarly, we offered $300 to each control group classes for their participation in the study. Training and Support. Teachers in the treatment group have to read the instructions for this experiment very carefully and act accordingly. Therefore, detailed trainings will be provided to the treatment group teachers. They will be instructed how to review and interpret the results of pretest, two interim tests and posttest and how to adjust their instruction based on the feedback they receive from the test outcomes.
The pretest, interim test and posttest rubrics also included descriptions of required and optional explanations such as: “A sufficient explanation includes a statement of fact such as: ‘Both lines are not proportional’ or gives a reason such as: ‘Because the graph does not go through the origin.’” Also optional explanations and calculation methods were shown. These error analysis and next step tools were designed to assist treatment teachers to utilize students’ answers to formal assessments to use formal assessments for formative assessment purposes. The error analysis is supported by mathematics education research experts (Alexander, 2013). Sample items from the pre test rubric with error analysis and next steps are shown in the accompanying attachment.
The main dependent variable in this study is the mathematics posttest and the main independent variable is the treatment group membership (treatment vs. control). Therefore, the principal analysis would be comparing the performance of treatment and control groups in the mathematics posttest. However, there are many other variables that may explain the performance difference between the treatment and control groups. First, in spite of random assignment of subjects to treatment and control groups some initial differences may remain. Therefore, some of the variables such as students’ pretest, their prior year state test scores can be used as covariates. Due to multivariate nature of this study and due to the fact that there are multivariate covariate and multivariate dependent variables, the analyses will be based on latent variables modeling rather than single structure variables. The use of latent variable helps to have a more reliable and valid measures by focusing on the common variance between groups of variables that are highly correlated. As mentioned earlier in this report, different instruments (such questionnaires for students’ background variables, fidelity of implementation of treatment, etc.) are used in this study. These data will be used as conditioning variables in a series of confirmatory factor analyses. Also, a pretest/posttest comparison is not used due to the content differences. The pretest assesses prerequisite standards knowledge and the posttest assesses summative content knowledge of the 8EE units of instruction. For example, the pretests may indicate student aptitudes and could be applied to both UCD (formal test treatment) and UCLA (task treatment) students. By design, the pretest is a rich source of formative assessment information for teachers in UCD treatment classrooms. |