CHAPTER NINE

DATA FROM OBSERVATION

S uccessful observation requires something more than just sitting and watching. (Lynch, 1996, p.
108)

In this chapter you will learn the difference between ordinary observation and research
observation, the advantages and disadvantages of classroom observation, what observer roles
can be taken, and some observation techniques. When addressing readers, I use the term TREE,
she, or you.

Preview Questions

1. Do you believe that “seeing is believing?” (Or, is its converse, “believing is seeing” true
sometimes as well?)

2. Have you ever seen something that turned out not to be what you thought?

3. What are some aspects of the second language classroom that can be observed?

4. What are some aspects that cannot be observed?

5. Have you ever observed any aspect of teaching and learning that you thought was
interesting and potentially worth researching?

Introduction

Even though observation is a basic source of human knowledge going back to Aristotle and
Herodotus (Adler & Adler, 1998), ordinary observation is not always a reliable source of
information. Eyewitnesses are often wrong when observing action, even at close range and in
good light. Here is a well-known experiment demonstrating this point: A researcher is lecturing
a class (often on the topic of observation), when suddenly two people making a lot of noise enter
the room wearing dark clothes. In one scenario, a man enters the room with a knife in his hand
being chased by a woman with a gun. After a brief time with much yelling, the man runs out of
the room followed by the woman. Immediately, the researcher asks the class to take out a piece
of paper and write down what just happened. In many cases, the gun-carrier is changed to the
man, and it is the woman who is being chased. This experiment is taken to illustrate that not only
are our observations not reliable, but that basic facts can be changed to fit our cultural and social
biases. We often see what we expect to see.

In spite of this shortcoming, observations can be used for research data collection (Nunan, 1992),
and also for teacher training (Day, 1990; Wang & Seth, 1998; Williams, 1989). However, these
two uses of observation (research vs. teacher training) are not closely related (Cook, 1989, p. 72).
This chapter describes observation for research data collection, drawing from four literature
sources: applied linguistics, evaluation, education, and anthropology. Applied linguistics
literature discusses observation from a classroom research point of view; evaluation literature

_________________________________________________________________________________
177 Data from Observation

 looks at educational programs, but also other types of programs such as medical, drug treatment,
business training, and government programs; education literature is primarily concerned with
conducting K-12 curriculum research; and anthropology observes people in their local situations.

Classroom observation defined

Ordinary observation tends to be sporadic, random, and accidental. For example, we are walking
down the street and see something happening in the park across the street; we are teaching a
class and notice something interesting; we walk by the open door of our colleague’s classroom and
hear something that gives us pause. These are all examples of observation, but they are random
and unplanned. On the other hand, research observation—including but not limited to classroom
research—must be systematic, intentional, and theoretical. Systematic means the observation is
not occasional, but must be principled so that it covers the area or time of interest (Genesee
& Upshur, 1996, p. 85; Hitchcock & Hughes, 1995, p. 234). Intentional means the observer has
a reason for observing. These reasons may be specific or vague, but the observer must have
something in mind when observing (Adler & Adler, 1998, p. 80). The reason or reasons for
observing can, in principle, be thought of as research questions or emerging research questions.
Theoretical means the observer is working with or looking for underlying principles. In fact,
Long (1980, p. 12) asserts that an observational instrument implies a theoretical claim about
second language learning and teaching. Stern (1989, p. 208) opines that by ignoring theory, we
run the risk of being overwhelmed by details that we can’t explain. Putting these ideas together,
we can define observation as the systematic, intentional, and principled looking, recording, and
analysis of the results of our observation for the purpose of research.

Consider again the example of the observation experiment mentioned above. Students were
sitting in a class when the incident happened. Their reports were not reliable, and thus not valid,
because the observers were not paying attention to the incident. Therefore, they could not have
any principled awareness, much less a systematic way of observing. But let’s imagine that at
least one of those students is again sitting in the classroom or maybe even teaching the class.
The chase happens. During the next class, it happens again. The observer thinks, “I should pay
attention to this.” The next time it happens, our observer takes careful notes, interviews others
to see what they believe happened, and maybe prepares to video the class hoping to record the
event. Now we can argue that these observations have, or are beginning to have, a fairly high
degree of reliability and validation, and could constitute observation data for research purposes.

What form might classroom observation take?

Classroom observation for research generally takes three forms. First, the TREE can observe her
own class or the class of another colleague. If the observation takes place in a class taught by
another teacher, the observer has more time to observe and record the results of the observation.
If the observation takes place in a class taught by the teacher herself, then there may be less
time and opportunity to record the events using paper and pencil for notetaking. This means the
physical recording of the data may occur at a later date. Second, the observation items may be
open or closed. Open means the observation item does not specify in advance what to look at or
record. Closed means the items are specified in advance. Open means the TREE is interested in

_________________________________________________________________________________
178 Data from Observation

what is happening, but has not determined exactly what she or he is looking for. Closed means
the TREE has decided what she or he is looking for. Third, the data gathered may be quantitative,
for example, frequency counts, or qualitative, for example, verbal descriptions. These questions
can help you decide which observation technique to use: Are you observing another teacher’s
class or are you observing your own class? Do you know what you are looking for (closed), or
are you starting with no predetermined categories (open)? Do you prefer data in numbers or
words? Table 1 shows how the answers to these questions provide eight possible categories of
observation.

Table 1. Eight forms classroom observation might take

____________________________________________________________________________________________________________
Orientation Observation Collecting data consisting of
category Numbers Words
____________________________________________________________________________________________________________
Observing Open 1. Things you notice 2. Initially you were not
another you can count, but were looking, but as the course went
teacher’s not looking for when on, you decided to look
class you started your at this more closely.
research.
____________________________________________________________________________________________________________
Closed 3. This is structured 4. Noting and describing high
observation, for example inference categories such as,
a checklist with are teaching tasks communicative?
predetermined
categories. You observe
and check the number
of times you see an
example of the category.
____________________________________________________________________________________________________________
Observing Open 5. Noticing something 6. Observing a class with
your own countable or fixed in no predetermined
class space or time. category. For example,
For example who audio recording
comes in late and by a class, and analyzing
how many minutes? it later.
____________________________________________________________________________________________________________
Closed 7. This involves 8. Using preset categories
recording your data to observe students and
using numbers or recording what you observe
ticking a box or using words. An example
category every time you could be noting how students
notice a predetermined perform a certain task.
event.
____________________________________________________________________________________________________________

_________________________________________________________________________________
Griffee 179
 What are the advantages of observation?

First, a researcher can directly observe a program (Rossi, Freeman, & Lipsey, 1999, p. 183) which
means she can focus on areas of interest including specific issues, materials, activities which can
be observed and evaluated (Fradd & McGee, 1994, p. 43). For example, when I was teaching in
Japan, some of the local Japanese high school teachers claimed they used English extensively in
their classrooms. I observed a class and found that the only use of English was about 8 seconds
when the teacher read a few sentences from the board. In addition, if one teacher is observing
another teacher’s class, observation data allows an understanding of the program not possible
by relying only on interviews (Patton, 1987, p. 12). Observation can give a “you-are-there” point
of view to readers not possible from other types of data (Patton, 1990, p. 203).

Second, a TREE can search for categories of research and evaluation that make sense to
participants (Alder & Alder, 1998, p. 89), or can be more focused by using predetermined
categories (Allwright & Bailey, 1991, p. 3). Predetermined categories can result in numbers that
can be treated statistically and analyzed (Hitchcock & Hughes, 1995, p. 235).

Third, since observation can be done by a teacher of her own class, students are not threatened
by outside observers, and can do their best work (Fradd & McGee, 1994, p. 43).

Finally, we have an advantage combined with a possible disadvantage. While observation data can
be rigorous when combined with other types of data (Alder & Alder, 1998, p. 89), to understand
what tallies, numbers, or checkmarks mean, it may be necessary to interview the teacher and
maybe the students to understand what they tallies or checkmarks mean (Hitchcock & Hughes,
1995, p. 238; Stern, 1989, p. 211). This is because a list of numbers derived from observation do
not interpret themselves. Observation may show us what is going on in a classroom, but not why.

What are the disadvantages?

There are several disadvantages to observation. First, not everything can be observed (Patton,
1987, p. 12). We can see only the surface of things (Richards, 1998, p. 142). Second, maintaining
observer openness is difficult because TREEs are familiar with teaching and classrooms (Bernard,
1994, p. 149). In anthropological fieldwork, for example, often the anthropologist visits the
village as an outsider and observes. Teachers, on the other hand, are not outsiders; rather, we
are villagers observing our own village. Because of their familiarity with classrooms, teachers
have to be trained, or train themselves, to note and question the obvious rather than take it for
granted and perhaps for that reason, not see it.

Third, observer education is not easy to learn and difficult to implement. Gebhard, Hashimoto,
Joe, and Lee (1999) note that in graduate level courses, when observing a class, students trained
to make specific comments such as “I saw the teacher write new vocabulary words on the board,”
regularly fell back to using vague and general comments such as, “it was a good class.” As a result,
if observers are not well trained, they tend to overestimate student performance (Fradd & McGee,
1994, p. 43).

_________________________________________________________________________________
180 Data from Observation

Fourth, if the observer is not the usual classroom teacher, the threat of research effect is always
present (Allwright & Bailey, 1994, p. 3). This means that an outsider observer can cause usual
classroom behavior to change. In addition, it is often hard for a teacher to find somebody to
observe her class (Fanselow, 1990, p. 191).

Fifth, observer bias is always present, since we all see and interpret events through the lens of
our own experience, assumptions, and interests. Another observer may notice something we do
not, or give a different interpretation. In addition, there is a high level of dependence on observer
articulation (Alder & Alder, 1998, p. 104; Allwright & Bailey, 1991, p. 3; Hitchcock & Hughes,
1995, p. 235). In other words, some teacher-observers may be quite clear and articulate about
what they see whereas other teachers may see the same thing and not be able to discuss it well.
Data from observation does not explain itself.

Sixth, interpretation based on observation data and analysis is not adequate for most research
purposes. In estimating student improvement, observation alone is usually not sufficient, and
other data collection instruments, such as tests, will be needed (Fradd & McGee, 1994, p. 216).

What are some of the key issues to consider?

One key issue is the role the TREE plays in observation. Long (1980) describes two situations
an observer may take: unstructured participant observation and nonparticipant observation.
In unstructured participant observation, often known as ethnography, the observer does not
set out to test a particular hypotheses, but takes a regular part in the activities of the group. A
nonparticipant observer, on the other hand, sits in the back of the room taking notes, and does
not take part in the activities. Vierra and Pollock (1992, p. 224) also describe two roles they call
participant observation and nonparticipant observation. Again, a participant observer takes an
active role in the activities whereas the nonparticipant sits in the back of the room and takes notes.
Patton (1990, p. 208) draws a continuum with detachment at one end and full participation at
the other. I accept Patton’s continuum and adopt Bernard’s (1994, p. 137) categories: detached
observer, participant observer, observer participant, and full participant.

More detached More involved

Detached Observer Participant Observer Observer Participant Full Participant

Figure 1. Observation participation options from detached to involved.

Detached observation would entail either observation through a one-way mirror, or having
somebody video the classroom in order to watch the video later. Participant observation occurs
when the observer is not a student and not the teacher. The participant observer would visit

_________________________________________________________________________________
Griffee 181
 the class several times so that her presence was not disturbing, but sit at the back of the room
taking notes. Observer participation occurs when the evaluator is much closer to the class. An
example of observer participant would be where the evaluator/researcher is also the teacher.
The fourth category is full participant, or what Bernard (1994) calls going native. An example
of full participation would be the researcher/evaluator being a student in the same program,
for example a graduate student in a seminar evaluating the seminar for the class project. Patton
(1990, p. 208) says deciding which role we play should be based on what is possible and what is
helpful. The point is that a TREE has several options when it comes to observer roles.

Another issue is the validation of observation techniques. TREEs in their teacher roles are
accustomed to many of the observation techniques, but may forget that when using observation
data for research, they must also present evidence that some care has gone into the collection
and analysis of the data collected. To assist TREEs in the validation of observation data, each
technique has validation suggestions.

Following are eleven observation techniques. Each is described, positive and negative aspects
are noted, and a validation plan is suggested.

1. Audio Recording Audio recording is the recording of all or part of a regular class. Procedures
for recording may depend on the level of technology available as well as the number of students.
An audio recorder with a microphone could be placed in a central location. For example, the
audio player could be placed on the floor and only the microphone attached by long cord placed
on a table or stand. Alternatively, small lapel mikes (called lav mikes) can be pinned on each
student or placed around their neck on lanyards. Another option is to place a small hand-held
recorder in a central location (Burns, 1999; Day, 1990).

Positive aspects: While some recording equipment can be bulky, a small hand held recorder is very
convenient. Once the recorder is switched on and functioning, one can concentrate on teaching,
knowing that data is being collected. A recording picks up linguistic details that otherwise may
go unnoticed. You can decide to listen to the recording only, or you can transcribe it, resulting in a
transcription others can read and study. Because audio recording produces a permanent record,
you may be able to count certain details and analyze them statistically.

Negative aspects: Audio recording may work for a class size up to 20, but what about a class
size of 40? Whole class activity is easier to record than group or pair work, although this could
be dealt with by providing one recorder to each group. Equipment of any kind, even a small
hand held recorder, is intrusive and may disconcert students so that what is recorded does not
represent typical behavior. Also, audio recording alone does not show us body language or other
contextual features. Transcription is time consuming, and when listening to a recording, it can
be hard to determine who is speaking. Finally, most recording devices have a limited recording
range, and anything out of that range is muffled or not understandable.

Validation plan: Describe your recording schedule so the readers of your report can see how
often and what days of the week you recorded. Tell them your recording procedures. What did
you do first, second? Describe your equipment. Where did you put the microphone? How did
your students seem to respond to being recorded? Did you tell them in advance? How did they

_________________________________________________________________________________
182 Data from Observation

respond when you told them? About how many minutes did the recording last each time? What
were you looking for? Is there any reason to believe that your students knew what you were
looking for? If there were, you run the risk of their responses not being typical. On the other hand,
maybe it didn’t make any difference if they knew or not. If you think it didn’t make any difference,
tell us why. How did you define what you were looking for? Were the data expressed in numbers
or words? If the data can be counted, what kind of reliability are you prepared to report? If
you used raters, tell us how you selected them. Generally, you don’t have to go into detail--just
a sentence or two like, “Two fellow teachers not directly associated with this evaluation were
selected as raters.”

2. Checklist A checklist is a form with predetermined or closed categories, usually listed down
one side of the page. Space is provided (often in little boxes) to mark the presence or absence of
the predetermined category. The resulting data are frequency data. The observation task can be
to check a yes/no category, or it can be to check or tick a box on a continuum (Day, 1990, p. 47;
Fradd & McGee, 1994, p. 272; Genesee & Upshur, 1996, p. 86; Richards, 1998, p. 144; Rossi, et al.,
1999, p. 226.

Positive aspects: A checklist is simple to make and easy to use. It can capture details of the lesson
you are evaluating. Low inference data, meaning the action observed by the evaluator does not
require much inference, can give reliable data. Data from checklists work well with other data
collection methods. For example, you could audio record group work activity, and later use a
checklist to collect data, providing a written record that can be easily stored for retrieval and
analysis. This way, the observed data are organized and easy to analyze.

Negative aspects: A checklist works well with low inference categories but not as well with high
inference categories. As mentioned, low inference means the action the evaluator is observing is
clear and easy to understand, and not much inference is required (e.g., How many questions did the
teacher ask?). Its opposite, high inference, means the action requires considerable interpretation
(e.g., Is the native language of the students valued by the instructor?). High inference categories
are hard to determine with a simple yes/no checklist item, and require corroborating data. Low
inference data may be reliable, but may lack a clear connection between what is observed and
what is measured. Unless a TREE records what is observed, she has no way of going back to the
original situation to verify the observer’s decision. In other words, readers have to accept the
researcher’s word. The key to using a checklist is being able to establish a clear link between the
predetermined category, the observed activity, and research questions.

Validation plan: It’s important to report how many categories were used, and what they were. In
an appendix, you can give the actual checklist form so the reader can see how it was arranged.
Tell readers why you decided to use those categories. How do you define each one? Did you
show your categories and definitions to a colleague for his/her critique? How many colleagues?
Did you ask them if any of the categories overlapped? How did they respond? If they offered
suggestions, what did you do? How many times did you administer the checklist? It could be
important to mention this, because if you administered it five or six times, it would probably
give more consistent results than if you did it only once. What happened when you first used
your checklist? Was it easy? Do you think you left anything out? If you think that happened, tell

_________________________________________________________________________________
Griffee 183
 your readers why. Did you conduct a pilot study with a group of students similar to those in your
study to familiarize yourself with how the checklist worked? This pilot data can be reported as
validation information. Piloting always strengthens your validation position. Report what you
did for your pilot, what you learned, and what changes you made as a result. Were your final
results supported by any other data collection techniques? If so, state which ones.

3. Teacher Hand Held Digital Recorder Hand held digital recorder (or the older technology
but sometimes available tape recorder) means holding a small, battery powered recorder, and
speaking directly into it. The purpose is to record observations during or soon after the time of
observation. The digital or taped record can be listened to or transcribed later (Burns, 1999, p.
88).

Positive aspects: These electronic devices are available and are relatively inexpensive to buy.
They are small enough to fit into pocket or bag. They use batteries, allowing freedom from cords
and plugs. Some tape recorders use a standard size cassette that can be played back on a larger
set when listening or transcribing. By holding it close to your mouth, the resulting recording
can be loud and clear. Your recorded notes are private--for your use only, or they can be shared.
Audio recording allows a permanent record to be made at the time of observation or soon after,
which means the audio notes may contain details that might have been forgotten as time passed
between the classroom observation and recording your observation. The recording can be played
back anytime, for example, in your office, or listening in a car while driving home. You can use the
recording as the basis for writing a more detailed, written record.

Negative aspects: Some teachers are not comfortable talking into a recorder, even in private. This
discomfort is magnified by using the recorder in public, for example, during class, after class, or
while walking down the hall. Also, sooner or later, recording equipment can fail, or batteries may
die and you won’t have extra batteries available.

Validation plan: The raw data is made up of the recording of yourself talking about what you
observed in class. This means you have anecdotal or unanalyzed data. Your words may contain a
mixture of descriptive data (what you saw) and evaluation or interpretation data (how you felt
about what you saw). Behind your evaluation/interpretation lie values. To identify a value, ask
yourself, “What would I have to value in order to feel about X the way I do?” Behind your values
is your teacher theory. You can use your raw data to unravel description from interpretation, and
then from the interpretation, to reveal your values and your personal theory. This would enable
you to understand not only how you feel, but why.

Another validation approach is to listen to your recording and take notes, or listen and do a whole
or partial transcription. A transcript provides raw data, which may be read and coded. Coding
is reading the transcript and deciding what themes are present, also known as units of analysis.
List the relevant data under each code. This process is called data reduction because any raw
data not directly relevant to a code is disregarded. Working with your codes and data, list your
interpretations. Writing up the process of going from your transcription to your interpretation
allows your readers to trace your steps, and this constitutes validation evidence. Again, the idea
is to document how you went from the raw data to an interpretation.

_________________________________________________________________________________
184 Data from Observation

A third validation approach is to listen to your recording several times and reach conclusions.
These conclusions are considered weak and unsupported because the readers have to take your
word for it, and can reasonably ask, “Why we should believe you--couldn’t you be mistaken?”
You can strengthen your position by stating how many times you listened to the recordings, that
you were able to get a peer to also listen to them, and that your peer either agreed with your
interpretations or came up with similar results on her own.

4. In-Class Observation Notes In-class observation notes refer to any type of written
documentation made by an observer other than the teacher while the class is meeting. Notes
written after the event are referred to as a Teacher Diary. Much has been written to describe
the strong and weak aspects of this type of observation documentation, which is a mainstay
of ethnographic observation (Adler & Adler, 1998; Allwright & Bailey, 1991; Bernard, 1994;
Evertson & Green, 1996; Fradd & McGee, 1994; Genesee & Upshur, 1996; Gebhard, 1999; Guba &
Lincoln, 1989; Lynch, 1996; Patton, 1987, 1990; Rossi, et al., 1999; Vierra & Pollock, 1992).

Positive aspects: The evaluator can observe an authentic educational environment, for example,
a classroom. Course goals and objectives can be verified. The observer does not have to rely
solely on handouts or secondhand reports of what is happening. Specific materials, activities, and
procedures can be observed and evaluated. Aspects of the course the teacher might miss, ignore,
or take for granted can be observed and noted. Observation data can be words or countable items
of interest, such as who speaks or where people sit. Nonverbal communication and behaviors,
for example, the way people dress or the way they use physical space can be observed. Aspects
of the classroom that might be meaningful, but are often overlooked--the bulletin board or room
decor--might be observed and noted.

Negative aspects: Observation using in-class notes is labor intensive. In addition, not everything
can be observed in a clear and obvious way. For example, high-inference categories, such as
student attitudes and feelings have to be carefully defined, and what you accept as evidence
carefully spelled out. Any instrument gives just one view. Observation data by itself is probably
not enough, and you may also need numerical, quantitative data, or qualitative interview data.
It is hard to learn how to be an objective observer and not mix descriptive observation with
evaluation and opinion. There are many sources of error, and Evertson and Green (1986, p. 183)
list seventeen of them, including problems with both rating scales and ethnographic observation.
Another consideration is that you need to negotiate an observation schedule with the teacher
that allows for observation on a systematic basis. Observing only a single class leaves you open
to charges of unreliability, that is, the claim that what you observed was an unusual occurrence
and only happened one time that semester, which happened to be the day you were there. If you
observe more than once, but you observe on the same day of the week, you run the risk of skewing
results because Friday classes may be different from Monday classes in small but important
ways. You also have to decide what role you want to play while observing. For example, while
observing, what would you do if the teacher or a student turned to you and asked a question?

Validation plan: If you have numerical data, see the validation plan for the observation technique
called Checklist. Assuming your data consists of words, ask yourself if your readers can follow
the actual sequence of how the data were collected, processed, and transformed for specific

_________________________________________________________________________________
Griffee 185
 conclusion drawing. Look at the validation strategy for Teacher Hand Held Digital Recorder to
see what you can apply. Tell your readers the context of your observation: How often and when
did you observe? Where did you sit? What was your level of class participation? What were the
students like? What did your notes look like? How did you organize them? How did you go from
observation to interpretations? What was your role in the class? In relation to the teacher, were
you a peer, a supervisor, trainer, friend, or some other role?

5. Peer Observation Peer observation refers to two teachers observing each other’s classes.
Both teachers typically adopt a participant observation stance, which means they sit in the back
of the room, observe, and take notes. A feedback session may be held later. Since they are peers,
evaluation of each other is not an issue. The purpose of peer observation from a course research
perspective is to gather evidence from a viewpoint not otherwise possible. Peer observation is
typically discussed in the context of teacher education (see for example Cosh, 1999; Fanselow,
1990; Gebhard, et al., 1999; Gebhard & Ueda-Motonaga, 1992; Richards, 1998, p. 147; Wang &
Seth, 1998; and Williams, 1989).

Positive aspects: Peer observation makes collegial research and evaluation possible. This means
that two teachers working together can research the same phenomenon, but in each other’s
class. Also, it may be possible to use another class as a mirror to evaluate an issue. For instance,
Teacher A can have an issue of interest and use observation of Teacher B’s class to illuminate that
issue. Teacher B may or may not even be aware of what Teacher A’s issue is.

Negative aspects: Peer observation is not common, so it may be hard to find a willing partner
because some teachers might see it as potentially threatening. It can also be difficult to arrange
schedules. The key to participating in peer observation is to have a clear idea of your purpose,
and for you and your peer to agree as to what you will do, when you will do it, and how you will
share information. More than likely, your data will take the form of written notes.

Validation plan: Refer to In-Class Observation and Teacher Hand Held Digital Recorder validation
plans.

6. Proformas As seen in Figure 2, a proforma is a grid or a form to fill in that can hold a type of
class performance, including descriptive as well as interpretative data (Burns, 1999; Day, 1990).

Time Period Category 1 Category 2 Category 3 Category 4

Figure 2. Example of a Proforma

_________________________________________________________________________________
186 Data from Observation

Categories of interest are written across the top of a piece of paper or card. Examples of column
headings could be Date, Question or area of interest, Descriptive Notes, and Interpretation.
Descriptive notes can be entered during or after class. If entered after class, the proforma
functions as a teacher diary, journal, or log. Interpretative comments could also be written after
class.

Positive aspects: A proforma can be an all-purpose data collection instrument. Using just this one
instrument, you can decide on a category, collect descriptive data, and engage in interpretation.
The key is the categories, which ultimately have to be explained. Deciding which categories fit
your situation may require trial and error to find the ones that give you helpful data. Questions
or categories can be predetermined; this preparatory activity helps to decide what to note.
Alternatively, categories can be relatively open and vague, depending on how clear you are about
what you are looking for. Proformas can be put on cards for easy filing. Proformas can be taken
into class, and there is space to jot notes on areas of interest. Proformas may take less time than
the Teacher Diary because notes are shorter and more to the point than diary or log entries.

Negative aspects: One problem is that if you are observing your own class, it may be difficult
to take notes while teaching. Another potential problem is forgetting to date the proforma, or
making notes so cryptic you can’t understand them later. Yet another problem is not interpreting
your data, so that later you have data that you don’t know what it means because much of the
context for the data is forgotten. A potential weakness of proformas is a tendency to use subjective
and ill-defined categories.

Validation plan: You should be sure to define the categories or at least explain what they mean. If
you are collecting verbal, descriptive data, use validation strategies similar to the ones discussed
in In-Class Observation and the three strategies discussed in Hand Held Digital Recorder.

7. Scribbles These are short and quick observation notes jotted down while teaching (Burns,
1999, p. 85).

Positive aspects: Notes made during the class have an authenticity of being made by someone
who was there and experienced a behavior or activity firsthand. Scribbles can be used as the
basis for recalling and writing more complete diary or proforma entries.

Negative aspects: It is difficult to teach (an active undertaking) and take notes (a reflective
undertaking) at the same time. It may difficult to keep scribbles in a systematic way. If you don’t
use your scribbles as the basis for more complete notes right after class, they may be hard to
read or understand later. Some teachers have trouble taking notes while teaching because they
tend to be peripatetic, moving around the classroom as they teach. On the other hand, it might
be possible to schedule brief periods of time for writing. Writing could be done in a small pocket
notebook, index cards, sticky notes, or in a text file on a mobile phone, tablet, or laptop.

Validation plan: Keeping a record of each time you gather scribbles could show your observations
to be intentional and systematic, two qualities that increase validity. As in Proformas, if you have
predetermined categories, you should tell your readers what they are, why you are interested in
them, how you selected them, how you define them, and how they are related to your evaluation

_________________________________________________________________________________
Griffee 187
 purpose and research questions. Probably the best idea would be to use scribbles as the basis for
another observation technique such as Teacher Diary.

8. Seating Chart A seating chart can be made either by an observer sitting in the back of the
room or by the teacher. It could be a single sheet of paper with boxes for each student showing
where each student is sitting. If you allow students to sit wherever they please, you might find
revealing patterns (Burns, 1999, p. 106; Day, 1990, p. 49).

Positive aspects: A seating chart could be helpful to your research if you are investigating a topic
such as student social relationships. While you are making a seating chart, it would be easy to
use the chart to incorporate other features of the room including the number and location of all
objects, such as windows and chalkboards, and perhaps the measurements and description of
the classroom. This information is helpful when it comes to writing your report because you can
describe the site with more detail. A seating chart can be combined with a video record to show
what happened and where.

Negative aspects: There are two kinds of data commonly provided by a seating chart: Where
students are sitting and a record of what students are doing, for example, how many times
each student initiates a speaking turn or other action. While sitting in the back of the room, an
observer can not only make a seating chart, but also have the time to indicate student interaction.
A classroom teacher can also make a seating chart, but may not be able to also conduct a class
while using such a chart to record interaction.

Validation plan: To validate seating chart data, you can create seating charts on multiple occasions
to show that seating patterns were consistent. Students tend to sit in the same place, but if for
some reason there is no consistency in where students sit, they this observation tool should be
not be used. If you draw conclusions about what certain seating patterns mean; theory, principles,
or prior research would be needed for explanatory purposes.

9. Structured Observation Structured observation is classroom observation using previously

defined categories. In some cases, an observation form is given to the observer with instructions to
note when, how often, or examples of classroom activities that in the observer’s opinion exemplify
the category. This implies rater training, a theoretical basis for the categories, and instrument
validation similar to that used by questionnaires and tests (Allwright, 1988; Chaudron, 1988,
1991; Cook, 1989; Fanselow, 1987; Galton, 1995; Long, 1980; Moskowitz, 1967; Nunan, 1992;
Spada, 1990; Spada, & Lyster, 1997; Weir & Roberts, 1994).

Positive aspects: There are many structured observation instruments available, for example Long
(1980) reviews 22 such instruments. Two of the better known are: FOCUS (Foci for Observing
Communications Used in Settings) (Fanselow, 1987); and COLT (Communicative Orientation
of Language Teaching) (Spada, 1990; Spada & Lyster, 1997). Despite criticism, there remains
continued interest in this type of data collection. When using low inference categories, relatively
objective and reliable data can be collected. Data collection may be recorded using audio or video
or in real time, and resulting data can be analyzed.

Negative aspects: The main danger is lack of validation evidence, that is, the chosen categories

_________________________________________________________________________________
188 Data from Observation

may exclude critical features from the data. The categories may appear relatively simple, but
even seemingly simple categories are often ambiguous and capable of multiple interpretations. In
addition, while one might find patterns, it does not follow that what is infrequent is insignificant.
Quantification of data into numbers cannot explain what those patterns meant to the participants.

Coding categories tend to focus on what a teacher says and does, which assumes that the teacher
controls classroom interaction. This may or may not be the case, for examplea structured
observation instrument may not be able to capture what goes on in pair or group work.
Concentrating on the teacher may also limit the usefulness of data from observation restricted
in the sense that nonverbal communication is not checked. Also, raters must be trained and
retrained from time to time. If an observer has a strong stake in the outcome, observer bias is
possible; the presence of an observer has some effect. Although observer effect can be overcome
to some extent by having neutral observers, being unobtrusive as possible, and having a sufficient
number of observations.

Since the data gathered are typically a count of certain features that provide frequency data, some
sort of statistical analysis and presentation is necessary. Those who aren’t trained in statistics
will consider this a weakness, while those who appreciate statistics will consider this a strong
point. The purpose of your research is key, and any observation instrument must serve the
research purpose. Much of the criticism of structured instruments comes from ethnographically-
oriented researchers who point out the problems of using predetermined categories. These
same criticisms can be made of any predetermined category instrument as well, such as tests or
questionnaires. For example, one charge is that using a predetermined category tends to focus
the observer’s attention in a specific direction, and in doing so, the observer may miss something
important that falls outside the category. The trouble with this criticism is that while it is true,
in fact, the point of focusing your attention is precisely that, to focus your attention. If you don’t
know what you are looking for, you shouldn’t use this type of instrument. But it is unfair to
criticize a predetermined instrument for being predetermined. It is like criticizing an apple for
being an apple and not an orange.

Structured observation instruments are the Rolls Royce of the observation world, and as such,
they are mostly used for research. Regarding the famous car, most of us have never driven--
let alone owned--one, and probably something similar can be said about using a structured
observation instrument. First, you have to find or adapt the one you want to use, then, you have
to be trained or train yourself to use it. Finally, you have to work out a validation plan. This type
of observation instrument may require finding a mentor experienced in its use, and also requires
familiarity with the literature. Be prepared to spend time in preparation, and be prepared to
conduct one or more pilots, in which you not only train yourself in the use of your instrument,
but also gather data for its validation. Essentially, you have three choices in using a structured
observation instrument: You can make your own, you can use an existing instrument, or you can
modify an existing instrument to suit your needs.

Validation plan: Before you even begin, create a log to record your procedures. Include as much
detail as possible; most teachers don’t gather enough of this information. The rater training
should be documented in detail, including who was selected, why, and how they were trained.

_________________________________________________________________________________
Griffee 189
 This is true even if you are the rater. Be prepared to deal with the charge of observer bias, since
the presence of an observer will have some effect. Inter-rater reliability is the most common as
well as the recommended form of reliability measurement. Another major issue is validation of
the categories. Validation involves establishing a link between the category, a theory or body of
knowledge that describes and defines the category, and your research purpose. You may want to
consider advanced statistical procedures such as factor analysis or structural equation modeling
as well as case studies to validate your data. See Weir and Roberts (1994, p. 173), who discuss
six validation strategies.

10. Teacher Diary A teacher diary, as used in this chapter, is a log or journal written primarily by
a teacher after a class session is over. This document is based on observations made during class
(see Bailey,1990; Bartlett, 1990, p. 209). When evaluating another teacher’s class, a teacher diary
serves primarily as a chronology of events and a repository for reflection. When evaluating the
TREE’s own class, this diary includes those two functions plus it takes the place of in-class field
notes. In this way, a teacher diary functions as a source for descriptive data, and as a source for
teacher reflective data.

Positive Aspects: A teacher diary is a term made popular in the applied linguistics literature
for ethnographic field notes, and as such, comes from a long and accepted history. It provides a
written document of what happened. If you don’t write it down, your recollection of events will
blur and merge with other memories until you can’t remember any details; details give your
report credibility.

Negative Aspects: Teacher diary data is recalled, rather than actually observed data. That means
the sooner you make your diary entry after the class, the better. It is easy to put off writing
(especially when you feel you have no time, tired, nothing special happened in class, or too many
other things to do when you get home). It is also hard to separate descriptive from evaluative
comments. Comments such as “Good class today” are evaluative rather than descriptive in that
they don’t tell readers what happened to make the class good. You have to train yourself to write
the details of what actually happened, as well as how you feel about it. Teacher feelings are
important, but they are not the same as detailing what caused them.

For a teacher evaluating another teacher’s class, a diary is helpful; for a teacher evaluating her
own class, a diary is indispensable. Without a diary, you run the risk of using only quantitative
data gathering instruments such as tests and questionnaires, which can give reliable and valid
data, but will make interpreting that data difficult. All data from your diary, even your reflections,
have to be considered raw data. That means all entries have to be analyzed and interpreted.

Validation plan: If you are observing your own class, your teacher diary can become a valuable
observation instrument, perhaps your only observation instrument. For that reason validation
must be carefully considered. It might be helpful for you to arrange your diary into four sections:
history, emotional reflection, descriptive data, and reflective data.

To offer validation evidence for your history, you should make frequent diary entries, even if
you have to force yourself. At the end of the evaluation period, write a history of the course as
completely as you can. Consider collecting documents, such as your local or school newspaper, to

_________________________________________________________________________________
190 Data from Observation

look for external events that might have influenced your students during the observation period.

To offer validation evidence for your emotional reflection, keep a record of your high and low
moods, what made you happy, depressed, elated, satisfied, and so forth. This can be an indication
as to your bias, or to put it more positively, your belief system. Try to match high emotional
states to events that caused or at least seem related to your high mood. Similarly, try to match
events to low moods. Can you state what might be your bias? If so, you can report this as part of
your validation data in that it helps your readers understand and interpret your findings. Do the
same thing for your teaching beliefs. How do events and bias indicate your beliefs? Your personal
theory is a filter through which all observations pass. It is helpful to report your understanding
of this so readers can take it into account.

To offer validation for your descriptive data, if you used Scribbles, Checklist, or Proforma, use
the validation strategies mentioned there. Describe in detail how you collected your data. When
did you write up your observations? Take readers through your process step by step so they
can follow and understand your data collection process. Search your data for themes. Code the
themes, that is, give each theme a name, and see if you can relate your themes to your research
questions. If you don’t have satisfactory research questions yet, try to use your themes to create
new ones or sharpen your present ones. Ask colleagues to look at your data, your themes, and
your research questions to see if they can follow your logic. If they can, report this as validation
data. If they can’t, work with your themes and RQs until they can.

To offer validation for your interpretations, ask a colleague to look at your interpretations (or
hypotheses). Are they plausible? If not, what additional evidence would you need? Find a critical
colleague who is willing to look at your data to see if they can find fault with your interpretation
and give you an “alternative hypothesis.” Don’t pick your best friend or someone who tends
to agree with you or share all your values. Finally, see if you can match your findings (your
interpretations) with findings from other sources of data, for example tests or interviews. This
data triangulation strengthens your findings.

11. Video This refers to the video recording of all or a portion of a class. The purpose of video
recording a class is to provide data to answer a research or evaluation question. For discussion
on the use of video see Burns (1999, p. 94), Day (1990, p. 46), Galton (1995, p. 502) and Woods
(1996, p. 37).

Positive Aspects: Small high-quality video cameras are now relatively inexpensive and available,
and many cell phones now include a video function. Video data can reveal things we might not
otherwise notice. Video can give a detailed naturalistic view of life in a classroom, and a sense of
being there.

Negative Aspects: Cameras don’t see everything, only what they are pointed at, and they can’t
be pointed at everything in a classroom all the time. You may have to buy the recording media,
get the camera, and aim the camera. Video gets harder as the number of students increases. If
you video a class for an hour, then you have to spend an hour watching the video, and perhaps
several hours transcribing the sound track, which may not always be of high quality. In addition,
video can be intrusive, and we run the risk of student reluctance to behave in a normal way in the

_________________________________________________________________________________
Griffee 191
 presence of a video camera, including students playing to the camera. There are certain ethical
considerations given the fact that students may be recognizable on the tape. You may have to
justify to an institutional research board (IRB) why you need to video students. If your students
are under 18 years of age, videoing without parental or guardian permission is likely illegal.
And yet, video is a technology that is here, can be used, and for some kinds of research, may be
required. For example, if interpretation of body language is part of a research question, a visual
record may be necessary. A school or department with video equipment, a budget, and staff to
operate the camera would be helpful. You might want to try a pilot study to see if video is helpful,
worth the trouble, and provides the kind of data you want.

Validation plan: If your data are countable, adapt the validation plan from Audio Recording. If
your data takes the form of words, look at how this type of raw data are validated in technique
number two, Teacher Hand Held Recorder.

In conclusion, the purpose of this discussion of observation techniques is to demonstrate that

many observation techniques are available, that they can be used alone or in conjunction with
others. Regardless of how they are used, they all must be validated.

Observation techniques grouped by use

In Table 2, the eleven observation techniques discussed in this chapter are grouped according
to the eight categories described in Table 1. The number in parenthesis behind each technique
is the number of its listing. Each of the eight groups of observation techniques is situated in a
particular research situation. If you are playing an outsider role, consult sets one through four.
If you are playing an insider role, evaluating or researching your own class, consult sets five
through eight. Decide if you have a predetermined category (you know what you are looking for),
here called closed, or if you do not have a predetermined category (you aren’t sure what exactly
you are looking for), called open. Decide which observation techniques fit your situation and best
answer your evaluation or research questions.

Observation techniques, both in the classroom and in other research situations, are time-honored
data collection instruments. However, observation of the type necessary for research and
evaluation requires a certain amount of attention that goes beyond what teachers typically do.
Not every observation technique fits every situation. A major difference seems to occur whether
you are observing another teacher’s class or your own. Sometimes validation reporting takes
the form of counting and statistical analysis, but often it does not. In many classroom research
studies, validation data of the data collection instruments are not reported. Whether we use
quantitatively or qualitatively oriented instruments, we are obligated to report how we got the
data, how the data were analyzed, how the data were interpreted, and what steps we took to
investigate the integrity of our data-gathering process.

_________________________________________________________________________________
192 Data from Observation

Table 2. Placement of Observation Techniques in Eight Categories

________________________________________________________________________________________________________
Use Observation Data collected in terms of
category Numbers Words
________________________________________________________________________________________________________
Observing Open 1. 2.
another Seating Chart (8) In-Class Observation (4)
teacher’s Peer Observation (5)
class
________________________________________________________________________________________________________
Closed 3. 4.
Checklist (2) In-Class Observation (4)
Seating Chart 8) Peer Observation (5)
Structured Observation (9)
________________________________________________________________________________________________________
Observing Open 5. 6.
your own Audio Recording (1) Audio Recording (1)
class Seating Chart (8) Handheld recorder (3)
Video Recording (11) Proformas (6)
Scribbles (7)
Teacher Diary (10)
Video Recording (11)
________________________________________________________________________________________________________
Closed 7. 8.
Audio Recording (1) Audio Recording (1)
Checklist (2) Handheld recorder (3)
Seating Chart (8) Proformas (6)
Video Recording (11) Scribbles (7)
Teacher Diary (10)
Video Recording (11)
________________________________________________________________________________________________________

Where can I read and find out more about observation?

Brian Lynch, in his book Language Program Evaluation (1996), includes a section on observation
as data gathering starting (p. 108). He also provides an example of a structured observation
instrument, the Communicative Orientation of Language Teaching (COLT) scheme. He describes
fieldnotes in detail, with a helpful discussion on the difference between descriptive as opposed to
evaluative comments. Anne Burns, in a book on action research, Collaborative Action Research for
English Language Teachers (1999), describes in detail many observation techniques, including
a discussion of validation. Weir and Roberts in Evaluation in ELT (1994), discuss observation
techniques. Finally, Genesee and Upshur, in Classroom-Based Evaluation in Second Language
Education (1996), devote an entire chapter to observation and provide a framework for classroom
observation.

_________________________________________________________________________________
Griffee 193
 DISCUSSION QUESTIONS

Write some questions you had while reading about observation.

Task 1. Describe a time when you observed something about your teaching or something that
happened in your class that you thought was interesting.

Task 2. How might your observation be framed as a research project?

_________________________________________________________________________________
194 Data from Observation

Task 3. Using the situation you first noticed in Task 1 and reframed as a research project in Task
2, what observation technique(s) described in this chapter could you use to gather research data?

Task 4. What additional data collection instruments might be helpful?

_________________________________________________________________________________
Griffee 195
 References for Data from Observation

Adler, P. A., & Adler, P. (1998). Observational techniques. In N. K. Denzin & Y. S. Lincoln (Eds.).
Collecting and interpreting qualitative materials (pp. 79-109). Thousand Oaks, CA: Sage.

Allwright, D. (1988). Observation in the language classroom. London: Longman.

Allwright, D., & Bailey, K. M. (1991). Focus on the language classroom: An introduction to classroom
research for language teachers. Cambridge: Cambridge University Press.

Bailey, K. M. (1990). The use of diary studies in teacher education programs. In J. C. Richards & D.
Nunan (Eds.). Second language teacher education (pp. 215-226). Cambridge: Cambridge
University Press.

Bartlett, L. (1990). Teacher development through reflective teaching. In J. C. Richards & D.

Nunan (Eds.). Second language teacher education (pp. 202-214). Cambridge: Cambridge
University Press.

Bernard, H. R. (1994). Research methods in anthropology: Qualitative and quantitative approaches

(2nd ed.). Walnut Creek, CA: Altamira Press.

Burns, A. (1999). Collaborative action research for English language teachers. Cambridge:
Cambridge University Press.

Chaudron, C. (1988). Second language classrooms. Cambridge: Cambridge University Press.

Chaudron, C. (1991). Validation in second language classroom research: The role of observation.
In R. Phillipson, E. Kellerman, L. Selinker, M. Sharwood Smith, & M. Swain (Eds.). Foreign/
Second language pedagogy research (pp. 187-196). Clevedon: Multilingual Matters.

Cook, V. (1989). The I-language approach and classroom observation. In C. Brumfit and R.
Mitchell (Eds.). Research in the language classroom (pp. 71-77). London: Modern English
Publications in association with the British Council.

Cosh, J. (1999). Peer observation: A reflective model. English Language Teaching Journal 53(1),
22-27.

Day, R. R. (1990). Teacher observation in second language teacher education. In J. C. Richards &
D. Nunan (Eds.). Second language teacher education (pp. 43-61). Cambridge: Cambridge
University Press.

Evertson, C. M., & Green, J. L. (1986). Observation as inquiry and method. In M. Wittrock (Ed.),
Handbook on research and teaching (pp. 162-213). New York, NY: MacMillan.

Fanselow, J. F. (1987). Breaking rules: Generating and exploring alternatives in language teaching.
New York, NY: Longman.

Fanselow, J. F. (1990). “Let’s see”: Contrasting conversations about teaching. In J. C. Richards & D.

_________________________________________________________________________________
196 Data from Observation

 Nunan (Eds.). Second language teacher education (pp. 182-199). Cambridge: Cambridge
University Press.

Fradd, S. H., & McGee, P. L. (1994). Instructional assessment: An integrative approach to evaluating
student performance. Reading, MA: Addison-Wesley.

Galton, M. (1995). Classroom observation. In L. W. Anderson (Ed.). International encyclopedia of

teaching and teacher education (2nd ed.). (pp. 501-506). New York, NY: Pergamon.

Gebhard, J. G. (1999). Seeing teaching differently through observation. In J. G. Gebhard & R.

Operand (Eds.). Language teaching awareness: A guide to exploring beliefs and practices
(pp. 35-58). Cambridge: Cambridge University Press.

Gebhard, J. G., Hashimoto, M., Joe, J., & Lee, H. (1999). Microteaching and self-observation:
Experience in a preservice teacher education program. In J. G. Gebhard & R. Oprandy
(Eds.). Language teaching awareness: A guide to exploring beliefs and practices (pp. 172-
194). Cambridge: Cambridge University Press.

Gebhard, J. G., & Ueda-Motonaga, A. (1992). The power of observation: “Make a wish, make a
dream, imagine all the possibilities!” In D. Nunan (Ed.). Collaborative language learning
and teaching (pp. 179-191). Cambridge: Cambridge University Press.

Genesee, F., & Upshur, J. A. (1996), Classroom-based evaluation in second language education.
Cambridge: Cambridge University Press.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

Hitchcock, G., & Hughes, D. (1995). Research and the teacher: A qualitative introduction to school-
based research (2nd ed.). New York, NY: Routledge.

Long, M. H. (1980). Inside the “Black Box”: Methodological issues in classroom research on
language learning. Language Learning, 30(1), 1-42.

Lynch, B. (1996). Language program evaluation. Cambridge: Cambridge University Press.

Nunan, D. (1992). Research methods in language learning. Cambridge: Cambridge University

Press.

Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.

Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA:
Sage.

Richards, J. C. (1998). Through other eyes: Revisiting classroom observation. In Beyond training.
Cambridge: Cambridge University Press.

Rossi, P. H., Freeman, H. W., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.).
Newbury Park, CA: Sage.

_________________________________________________________________________________
Griffee 197
 Spada, N. (1990). Observing classroom behaviours and learning outcomes in different second
language programs. In J. C. Richards & D. Nunan (Eds.). Second language teacher education.
Cambridge: Cambridge University Press.

Spada, N., & Lyster, R. (1997). Macroscopic and microscopic views of L2 classrooms. TESOL
Quarterly 31(4), 787-792.

Stern, H. H. (1989). Seeing the wood and the trees: Some thoughts on language teaching analysis.
In R. K. Johnson (Ed.). The second language curriculum. Cambridge: Cambridge University
Press.

Vierra, A., & Pollock, J. (1992). Reading educational research. Scottsdale, AZ: Gorsuch Scarisbrick.

Wang, Q., & Seth, N. (1998). Self-development through classroom observation: Changing
perceptions in China. English Language Teaching Journal, 52(3), 205-213.

Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell.

Williams, M. (1989). A developmental view of classroom observations. English Language Teaching

Journal 43(2), 85-91.

Woods, D. (1996). Teacher cognition in language teaching. Cambridge: Cambridge University

Press.

_________________________________________________________________________________
198 Data from Observation