FONDEF Project proposes innovative writing evaluation system for 5th grade students

The objective of the project was to build a test that had principles of equity and social justice in all dimensions of its design.

“Towards a socially fair evaluation of writing” is the name of the FONDEF Project that, after two years of development, presented its results on December 19 at the UC Faculty of Education. 

The project, led by the Faculty academic and director of Postgraduate Education UC, Natalia Ávila, sought to build a test that had principles of equity and social justice in all dimensions of its design. For this, a standardized test was created to be applied on a large scale. Unlike evaluations that are used only to provide information to the system, this one goes further, because it is associated with a teaching system, which serves as a large-scale diagnosis.

The research was led by the UC and developed in conjunction with the universities Diego Portales y Alberto Hurtado, who formed an interdisciplinary team of experts in writing and measurement, linked to public policiess. During the presentation of results, the alternate director of the project and researcher from MEASURE UC, Diego Carrasco, the researchers María Jesús Espinosa, Javiera Figueroa and Rosario Notary, and the researcher Ernesto Trevino. 

In the project, the Directorate of Public Education and the Research and Innovation in Education Laboratory, SUMMA. Also present during the event were the director of MIDE UC, Jorge Manzi, the head of the Subdepartment of Educational Programs and Professional Development of the Directorate of Public Education, Francisca Zamorano, and the dean of the Faculty of Education UC, Alejandro Carrasco. 

“This project does something very important, which is to try to decouple the effect that the socioeconomic context has from the school effect, or from the teacher effect or educational interventions. Decoupling that is very difficult and challenging, and doing it particularly about writing, like this project, seems very important to me. I think it is a great contribution, not only in Chile, but to global educational research,” said Dean Alejandro Carrasco during the activity.

The day featured a conversation panel led by María Ángeles Castillo, professor at Colegio Campanario; Tamara Rozas, Evaluation and Standards coordinator of the Curriculum and Evaluation unit of the Mineduc; and Paulina Videla, head of the Learning Achievement Assessment Division of the Education Quality Agency.

Absence of bias

At the meeting, Natalia Ávila presented a text prepared by one of the students who participated in the test: a pamphlet on the prevention of animal abuse, written by a fifth grade boy or girl. 

“In this text, which is brief, we can see several indicators regarding the writing capacity of this young writer. The text begins with a sentence in the first paragraph that introduces the topic and its importance: 'Animal abuse has been caused by humans', and then exemplifies. The theme is developed with two more paragraphs, each one with a clear theme,” explained Natalia Ávila.

“All of us here are literate adults, therefore we tend to underestimate the complexity of writing. But the truth is that writing is an incredibly complex activity,” the academic pointed out. “For the same reason, writing is also a very complex skill to measure. And it is not only a cognitive skill, but it is also a social practice.” 

The fact that it is a social practice, he explained, means that there are some forms that are more prestigious and socially valued than others. Forms of writing that are formal, academic, and closer to the “dominant habit.” A very important challenge when asking how to evaluate writing has to do precisely with how to untangle that complex network in which previous social experiences determine what is considered correct when writing.

“These results are very promising in the current context, in which new instruments are needed to measure possible delays due to the pandemic and in which the role of SIMCE and standardized tests in the country is being rethought. The project proposes concrete and innovative ways to advance to tests that are not biased and that benefit the system through the development of fair use of scores, such as teaching based on their results,” added Natalia Ávila. 

For the project, 6 studies were designed, 3 of them experimental. First, we investigated the topics and tasks that generated the most motivation for writing; Second, a device was tested that activates prior knowledge about the topic and the genre to be written, and that allowed students to increase their chances of writing better texts in the test by 2.48 times. In other studies, the rubrics and the most appropriate correction format were tested. The fifth study designed a teaching system associated with scoring. 

A teaching proposal associated with the test results consisting of two learning routes was tested with the support of teachers, where each one addressed different aspects of writing that allow the test results to be used to teach and continue improving. The final study found that the instrument is invariant to the parents' education, that is, it would not introduce a socioeconomic bias when measuring writing.