THE USE OF GUSKEY’S PROFESSIONAL DEVELOPMENT EVALUATION MODEL FOR EVALUATING LESSON STUDY PROGRAM A Paper presented at the 13 Indonesia Conference on Lesson Study, Bandung, August 26-29, 2022 th 1, 2 Harry Firman1, Nahadi2, Harun Imansyah3 Department of Chemistry Education, Indonesia University of Education 3 Department of Physics Education, Indonesia University of Education Correspondence address: harryfirman510@gmail.com Abstract: The increasing number of lesson studies implemented by the Government, donors, and schools to improve the quality of educational processes and outcomes in Indonesia requires the support of evaluation process that estimate the success level of its implementation. Program evaluation needs to be a complement to the implementation of the lesson study, because it will provide information about the real success level of the program being implemented, as a basis for creating continuous improvement of the implemented program sustainably. This paper highlights an evaluation design for lesson study program based on Thomas Guskey's model of evaluating professional development. Five-level of evaluation according to Thomas Guskey (participants' reactions, participants' learning, organization support and change, participants' use of knowledge and skills, student learning outcomes) will be elaborated in this paper and applied to evaluation design for lesson study program. The measure, data structure, data collection which are needed to evaluate the effectiveness of the lesson study program will be presented in detail. Key words: Lesson study, program evaluation, Guskey’s five level evaluation model, continuous improvement. INTRODUCTION Lesson Study (LS) was introduced to the world of education in Indonesia through a decade of JICA's series of projects, namely JICA-IMSTEP (1998-2003), JICA-IMSTEP Follow-up (2003-2005) and JICA-SISTTEMS (2006-2008) (Suratno, 2012). Sustainability of that JICA projects funded by UPI and Ministry of Education in cooperation with some local educational authorities as well as private sectors in Indonesia causing that currently more and more lesson studies are applied in the professional development of teachers at the school level and MGMP as an effort to improve the quality of learning at classroom level. 1 In Japan, lesson study is a culture of the teacher community in Japan, which embodied weekly regular meetings of teacher groups of similar subjects to reflect personal experiences of teaching to identify problems encountered in teaching, and collaboratively design, implement, evaluate, enhance innovative learning to solve the problems faced. The lesson study culture makes the process of continuous improvement steady in the quality of learning in Japan, which makes the competence of Japanese children at the top of the world as demonstrated by their performance in a series of international comparative studies, such as PISA and TIMSS. There is no information published regarding program evaluation of the lesson study in Japan. The views of some Japanese education experts confirm that there is no reason for carry out formal program evaluation of lesson studies carried out in schools in Japan, because the chain of evaluation and improvement is already part of the lesson study culture. However, for Indonesia, which is still in the early stages of lesson study enculturation, where lesson studies have only been implemented in a small number of schools and are still guided by external parties with diverse modes of its implementation, so that the impact of lesson study on improving student learning outcomes is uncertain (Suratno, 2012). Therefore, the evaluation of lesson study programs with a focus on each step of the lesson study process is actually very important. Program evaluation that aims to identify the needs of program participants, test whether the processes in the program meet the needs of participants, and measure program outputs (Posavac & Carey, 2007) are needed to provide information about the effectiveness of each step of implementation and the overall output of lesson study. On the basis of such information Government and other parties can make decision making to enhance the implementation process so that the lesson study purposes can be achieved. In accordance with the characteristics of lesson study as the professional development of teachers, the frame of reference for evaluating lesson study programs is more appropriate to refer to the approach of evaluating professional development than other evaluation approaches. METHOD The purpose of this study is to design evaluation method for evaluating lesson study program that is implemented in the early phases of lesson study adoption so that information is available to increase the effectiveness of each stage of the lesson study. With the developed evaluation design, it can be expected that the implementation of the entire lesson study process is effective to enhance student learning outcomes, as the ultimate goal of lesson study implementation. The first step of the study is to analyze the anatomy of the lesson study 2 program comprehensively, followed the second step which is the application of the selected professional development program evaluation model for evaluating each step of the lesson study. The result of this study is a program evaluation design that will available as a framework for planning and implementing the evaluation of the lesson study program. RESULT AND DISCUSSION Lesson Study as a School Based Professional Development Lesson study is part of the routine life of the teacher community in Japan which is manifested by weekly regular meetings of teacher groups of similar or related subjects to reflect on personal experiences teaching certain topics to identify problems encountered in teaching in their classes, and collaboratively find ways to understand the possible causes of these problems. Furthermore, the idea of a learning design that is considered prospective and feasible is planned in the form of a lesson plan and its teaching materials needed, and implemented by one of the teachers in a real class while observed by his colleague. The focus of observation is how students' learning behaviors when they engage in an enhanced learning process, and the results are discussed together to evaluate its successes and failures, to then determine the next steps to improve them. Stigler & Hiebert (1999) and Fernandez & Yoshida (2004) describes a cycle of the lesson study process in several steps, which includes: Step 1. Defining the problem. Teacher groups discuss a learning problem, discuss the causes of problems, as well as ideas for solving problems Step 2. Planning the lesson. Immediately after the idea of improving learning, the group collaboratively draws up a lesson plan and prepares teaching materials (e.g. worksheets, hand-outs or media). Step 3. Teaching the Lesson. One of the teachers carries out the learning according to the lesson plan, while the observer teachers sit in the back of the class at the beginning of the lesson, but when the student starts to have a discussion or group work, the observer goes around observing and taking careful notes about what the students in his group are talking about and doing during the lesson. Sometimes video recording is done so that learning can be studied then more carefully. Step 4. Evaluating the Lesson and Reflecting on Its Effect. After the learning is over, the teacher and colleagues who made the observations conducted a discussion. The first 3 opportunity in the discussion is given to the teacher who teaches to entangle his views on which plans work and what problems are encountered when carrying out learning. Furthermore, the observer teacher takes turns to critically state the part of the learning that he saw as problematic. The focus of the conversation is on learning, not on the teacher who teaches, so the discussion becomes a vehicle for self-criticism, for the purpose of improving the lesson plan made together Step 5. Revising the Lesson. Based on the reflection on the results of the observations made, the teacher group revised the lesson plan, and perhaps also replaced the teaching materials, student activities, problems posed to the students. The changes made are based on misconceptions experienced by students during learning. Step 6. Teaching the Revised Lesson. Once the revision of the lesson plan is completed, the learning is carried out again in other classes. Sometimes the same teacher carries out learning, but often other teachers carry out learning. Step 7. Sharing the Results. The results of one group's lesson study are shared to the wider community of teachers of similar subjects through several ways. First, write a report published by the school in a collection of lesson study results from various subject teacher groups, to be read by a wider audience. Another way is to be demonstrated to fellow teachers from other schools in the school's regular open-house agenda, to get criticism, views, or comparisons from lesson study groups from other schools. A specific pedagogy expert from university might be invited to deliver his/her review on the demonstrated teaching. From the entire lesson study steps described above, it can be concluded several main features of the lesson study, among other things: (1) Lesson study maintains focus on student learning consistently; (2) Lesson study focuses on direct improvement of teaching; (3) Lesson study is collaborative; and (4) Lesson study is based on long-term continuous improvement model; and (5) Teacher who participates in lesson study see their own teaching from a realistic and grounded perspectives, and make enhancement of their knowledge of teaching. These features ensure that lesson studies act as school-based professional development activities, which are not only for the purpose of increasing teacher competence, but also to improve the quality of the learning process. This conclusion is confirmed by Fernandez & Yoshida (2004) who stated that lesson study provides teachers with an opportunity to discuss the content that they are called on to teach and in so doing teachers can refine their understanding on this 4 content, understanding of how students think and learn, work with other teachers to develop their pedagogical knowledge and skills. Guskey’s Model for Evaluating Professional Development Program There are various models offered to be a framework for program evaluation, but what really needs to be considered is the suitability between the program evaluation model and the evaluation purposes and the program specifications to be evaluated. One program evaluation model that is specifically for evaluating professional development is the evaluation model initiated by Thomas Guskey, which is more suitable to be applied to the evaluation of lesson study programs because lesson study is a teacher professional development program. Guskey (2010, 2016) states that effective evaluation for teacher professional development needs to consider five stages of critical information. Guskey further stated that the five levels in this model are hierarchically arranged, from simple to more complex. With each succeeding level, the process of gathering evaluation data requires more time and resources. And because each level builds on those that come before, success at one level is usually necessary for success at higher levels. Level 1 (Participants’ reaction) The first level of evaluation looks at participants’ reactions to the professional learning experience. At this level, questions focus on whether participants liked the experience. Did they feel their time was well spent? Did the content and material make sense to them? Were the activities well-planned and meaningful? Was the facilitator knowledgeable, credible, and helpful? Did they find the information useful? The participants’ initial satisfaction provides information how to improve the design and facilitation of professional development. Positive reactions from participants are usually a necessary prerequisite to higher-level evaluation results. Level 2 (Participants’ Learning) Level 2 focuses on measuring the new knowledge, skills, attitudes or dispositions that participants gain. Although Level 2 evaluation data often can be gathered at the completion of a professional development program, it usually measures that reveal attainment of specific learning goals, so that performance indicators of successful learning should be prepare before activities begin. If there is concern that participants may already possess the requisite knowledge and skills, evaluators may require some form of pre- and post-assessment. 5 Analysing this data provides a basis for improving the professional learning’s content, format, and organization. Level 3 (Organizational Support and Change) At Level 3, the focus shifts from participants to organizational dimensions that may be vital to the success of the professional development. Organizational elements also can sometimes hinder or prevent success, even when the individual aspects of professional development are done right. The lack of positive results in this case does not reflect poor training or inadequate learning on the part of the participating teachers, but rather organizational policies that are incompatible with implementation efforts. Problems at Level 3 have essentially cancelled the gains made at Levels 1 and 2. Level 3 questions focus on the organizational characteristics and attributes necessary for success. Did the professional development promote changes that were aligned with the mission of the school? Were changes at the individual level encouraged and supported by the headmaster? Were sufficient resources made available, including time for sharing and reflection? Were successes recognized and shared? Issues such as these often play a large part in determining the success of any professional development. Structured interviews with participants and school administrators are required to collect of these data. Level 4 (Participants’ Use of New Knowledge and Skills) At Level 4, the primary question is: Did the new knowledge and skills that participants learned make a difference in their professional practice at classroom level? These data cannot be gathered at the end of a professional development program. Enough time must pass to allow participants to adapt the new ideas and practices to their settings. The most accurate data typically come from direct observations, either by trained observers or using digital recordings. Analysing these data provides evidence on current levels of use. Level 5 (Students Learning Outcomes) Level 5 addresses the bottom line in education: What was the impact on students? Did the professional development benefit them? The particular student learning outcomes of interest will depend on the goals of that specific professional development endeavour. Measures of student learning typically include cognitive indicators of student performance and achievement, such as assessment results, portfolio evaluations, marks or grades, and scores from standardized tests as well as affective and psychomotor or behavioural indicators of student performance. 6 Design of Lesson Study Program Evaluation Guskey's model of evaluation is applicable for any kind of professional development, including lesson study. However, in applying this evaluation model to the lesson study program, it is necessary to contextualize the structure of the lesson study program that is carried out. Normally in Indonesia, lesson studies are introduced to teachers through government programs or the private sector in collaboration with educational universities, as a form of local subject teacher working group lesson study (type 1) or school-based lesson study (type 2). In type 1 lesson study participants were teachers from a number of schools who taught the same subject, whereas in type 2 lesson study, lesson study activity is carried out in a school that involves all subject teachers. The duration of the program is generally 1 year which includes 3 cycles of lesson study activities. Each lesson study activity cycle begins with an introductory explanation of the lesson study in one day session. The next activity is divided into three phases, namely the PLAN phase (defining the problem and lesson planning) in around two weekly meetings, the DO phase (teaching and observing the lesson) and the SEE phase (reflection and evaluation) in one full day session. Evaluation of a lesson study program by referring to the Guskey’s program evaluation model is designed in accordance with the structure of the lesson study implementation. The evaluation design of the lesson study program is shown in Table 1 which describes the evaluation objectives, the required data/information, types of instruments that need to be developed for each level of evaluation as well as best timing of data collection. Of course, the program evaluation design for lesson studies carried out with difference strategies requires adjustments to the context. However, the completeness of the Guskey’s model of program evaluation needs to be maintained to provide comprehensive evaluative information about the lesson study program being evaluated, so that the enhancements policy of the implemented lesson study program can be well directed. 7 Tabel 1. Design of Lesson Study Program Referring to Guskey’s Model Stage of Evaluation Level 1 Participants reactions Required Data and Information Participant satisfaction with the LS activities carried out Evaluation Instrument Level 2 Participants Learning • Performance in participating in LS activities Guidelines for observing teacher activities in participating LS (PLAN, DO, and SEE) At the end of the PLAN and DO phase Lesson plan assessment rubric After the PLAN Phase • Teacher's ability to prepare lesson plans • Teachers' ability to develop learning tools • Teacher performance in teaching the planned lesson • Student learning outcomes • Student impressions of learning carried out by participation teacher Questionnaire of participant satisfaction with management, debriefing, each step activities in LS Teaching materials and learning media assessment rubric Teaching performance assessment rubric Quiz Students’ impression questioner on experienced learning Data Collection Time At the end of the PLAN-DO-SEE cycle After the PLAN Phase During DO Phase At the end of DO phase At the end of DO phase Level 3 Organizational Support & Change School management evaluation on implementation LS in their school, the support provided by the school, and changes brought about by LS participant teachers A guide to interviews with school management about lesson study program At the end of the SEE phase Level 4 Participants Use of New Knowledge and Skills Teaching performance of exLS participating teachers in daily teaching Teaching performance assessment rubric About one month after lesson study finished Academic test About one month after lesson study finished Level 5 Students learning Students achievement Learning Outcomes Note: LS is abbreviation of Lesson Study CONCLUSION This study resulted in an operational design of the evaluation of the lesson study program referring to Guskey's model of professional development program evaluation. This design allows the evaluation of the lesson study program to be carried out comprehensively and in 8 depth, hence providing relevant information for lesson study organizers to make corrective decisions and strengthen the effectiveness of the lesson study program. Another important implication is that the position of program evaluation is very important complement to the implementation of lesson studies in order to make lesson studies that are proven successful to improve the quality of learning in Japan as home countries will be successfully to improve the quality of learning in the countries of its adopters as well. REFERENCES Fernandez, C., & Yoshida, M. (2004). Lesson study: A Japanese approach to improving mathematics teaching and learning. Mahwah, NJ: Lawrence Erlbaum Associate. Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Guskey, T. R. (2016). Gauge impact with 5 level of data. Journal of Staff Development, 37(1), 32-37. Posavac, E. J., & Carey, R. G. (2007). Program evaluation: Methods and cases studies. Upper Saddle River, NJ: Pearson Education, Inc. Stigler, J. W., & Hiebert, J. (1999). The teaching gap: Best ideas from the world’s teachers for improving education in the classroom. New York, NY: The Free Press. Suratno, T. (2012). Lesson study in Indonesia: Indonesia University of Education experience. International Journal for Lesson and Learning Studies, 1(3), 196-215. 9