Design and Development Research

12
Rita C. Richey and James D. Klein

Abstract
This chapter focuses on design and development research, a type of inquiry unique to the
instructional design and technology field dedicated to the creation of new knowledge and
the validation of existing practice. We first define this kind of research and provide an over-
view of its two main categories—research on products and tools and research on design and
development models. Then, we concentrate on recent design and development research
(DDR) by describing 11 studies published in the literature. The five product and tool studies
reviewed include research on comprehensive development projects, studies of particular
design and development phases, and research on tool development and use. The six model
studies reviewed include research leading to new or enhanced ID models, model validation
and model use research. Finally, we summarize this new work in terms of the problems it
addresses, the settings and participants examined, the research methodologies employed
used, and the role evaluation plays in these studies.

Keywords
Design and development research • Instructional and non-instructional products • Design
and development tools • Instructional design models

advocating this type of research for the past 15 years (Klein,

The Empirical Nature of Design 1997; Richey, 1997; Richey & Klein, 2005, 2007, 2008;
and Development Richey, Klein, & Nelson, 2004; Richey & Nelson, 1996).
This chapter is a continuation of our work. It examines
Design models often parallel the scientific problem solving design and development research (DDR) by providing an
processes. Thus, the practice of design and development is to overview of its definition and scope. The major part of the
a great extent empirical by nature. Therefore, it would rea- chapter focuses on representatives of recent design and
sonable to assume that design and development processes development research. Finally, we summarize this new work
have robust empirical support. Yet historically there has been with special emphasis on the problems it addresses, the set-
a scarcity of research on our models, products and tools. tings and participants examined, the research methodology
While there has been increased empirical work on design used, and the role evaluation plays in these studies.
and development recently, we have been writing about and

Design and Development Research: Rationale,

R.C. Richey (*)
Wayne State University, Detroit, MI, USA
Definition, and Scope
e-mail: rrichey@wayne.edu
Opinions on the role of research on design and development
J.D. Klein
Florida State University, Tallahassee, FL, USA often depend on one’s own view of what it actually is. We
e-mail: jklein@fsu.edu take the position that design and development is a science,

J.M. Spector et al. (eds.), Handbook of Research on Educational Communications and Technology, 141
DOI 10.1007/978-1-4614-3185-5_12, © Springer Science+Business Media New York 2014
142 R.C. Richey and J.D. Klein

even though it is highly influenced by the creativity of the interests and endeavors that are not a part of this orientation.
designer. We approach design and development (and in turn DDR does not encompass the following: instructional psy-
research on it) with the assumption that science and empiri- chology or learning science studies; media delivery system
cism provide a more effective and reliable route to disciplin- comparisons or impact studies; message design studies; and
ary integrity than depending on artistic tactics and craft-based research on the profession. While results from research in
solutions. As a science, design and development should be these areas impact design and development, the study of
bound by understandings built upon replicated empirical variables embedded in such topics does not constitute DDR.
research. Our models and procedures should be validated. Design and development research, as with all research
The solutions to our problems should be supported by data. endeavors, leads to knowledge production, a more complete
We believe that our field has not sufficiently employed understanding of the field, and the ability to make predic-
empirical methods to facilitate our understanding of design tions. DDR reaches these goals through two main categories
and development processes. The need for research is espe- of research projects: (1) research on products and tools and
cially critical with respect to the models and processes (2) research on design and development models. We previ-
employed by designers and developers. Few models, design ously referred to these two categories of design and develop-
strategies, and tools employed in practice have been empiri- ment research as Type 1 and Type 2 developmental studies
cally tested and validated. This is the gap that design and (Richey, Klein, & Nelson, 2004). Others have referred to
development research seeks to address. instructional product development studies as design-based
Design and development research is a type of inquiry research (Wang & Hannafin, 2005), systems-based evalua-
unique to the Instructional Design and Technology (IDT) field tion (Driscoll, 1984), and formative research (Reigeluth &
that is dedicated to the creation of new knowledge and the Frick, 1999; van den Akker, 1999).
validation of existing practice. We define DDR as “the system- Below we describe design and development research in
atic study of design, development and evaluation processes detail and briefly examine 11 studies conducted since 2007
with the aim of establishing an empirical basis for the creation in this line of inquiry. In addition to being quite recent, these
of instructional and non-instructional products and tools and studies were selected to exemplify the major categories of
new or enhanced models that govern their development” DDR, the types of methodologies commonly employed, and
(Richey & Klein, 2008, p. 748). This definition aligns with the range of research settings examined. They also highlight
recent suggestions that professionals in the IDT field facilitate studies conducted in a variety of locales around the world.
learning and improve performance by creating, using and We begin by discussing research on product and tools, fol-
managing appropriate instructional and non-instructional lowed by an examination of research on models.
interventions (Definition and Terminology Committee of the
Association for Educational Communications & Technology,
2007; Reiser, 2012; Richey, Klein, & Tracey, 2011). Research on Products and Tools
Design and development research covers a wide spectrum
of activities and interests. It includes the study of the design The most straightforward type of DDR falls into the
and development process as a whole, of particular compo- first category—research conducted during the design and
nents of the process, or the impact of specific design and development of a product or tool. Often, the entire design
development efforts. Such research can involve a situation in and development process (analysis–design–development–
which someone is studying the design and development work implementation–evaluation) is documented. In some cases,
of others. It can also involve a situation in which someone is researchers concentrate only on one facet of design and
performing design and development activities and studying development (e.g., needs assessment). Many recent studies
the process at the same time. In either case, there is a distinc- focus on the design and development of technology-based
tion between doing design and development and studying the products and tools.
processes. Below, we discuss three classes of product and tool
Design and development research is an umbrella term for research. These include studies of (1) comprehensive design
a wide range of studies that employ an assortment of tradi- and development projects, (2) specific ID project phases, and
tional quantitative and qualitative research methods and (3) tool development and use. We review recent representa-
strategies. Most design and development research, however, tive product and tool research in each of these categories.
tends to rely more on qualitative strategies and deals with
real-life projects, rather than with simulated or contrived
projects. Many studies can be viewed as multi-method Recent Comprehensive Design
research. and Development Research
Understanding the nature of this research is a matter of
understanding the range of problems to which it can be Studies of comprehensive design and development proj-
applied. It is also a process of recognizing those research ects usually demonstrate the range of design principles
12 Design and Development Research 143

available to practitioners. Frequently, the entire design and employed multiple research methods, considered issues such
development process is studied and documented. The design as researcher bias and instrument reliability, and collected
processes used in a particular situation is described, ana- data from several sources.
lyzed, and a final product is evaluated. Consistent with
predominant practice in the field, the procedures employed A Task-Centered, Peer-Interactive Course Redesign
usually follow the tenets of instructional systems design A descriptive case study by Francom, Bybee, Wolfersberger,
(ISD), encompassing analysis through evaluation. This is the and Merrill (2009) provides another example of comprehen-
case in a research study by Visser, Plomp, Armiault, and sive design and development research focusing on a product.
Kuiper (2002) who describe the design and development of This study addresses the real-world problem of converting a
a product which addresses learner motivation in distance passive, face-to-face college biology course to an online
education programs. This work includes an initial pilot study, course that includes peer-interaction and task-centered
as well as a year-long try-out and evaluation of the product. instruction. The authors describe how the instructor selected
A study by Sullivan, Ice, and Niedermeyer (2000) is also an content topics and “complex, authentic tasks that would
example of comprehensive DDR that focuses on the impact require students to gain a sufficient knowledge of the subject
of an instructional program. These researchers use field eval- area in order to complete the task” (p. 37). They also explain
uation to test a comprehensive K-12 energy education cur- how the First Principles of Instruction (see Merrill, 2002)
riculum that was the product of a design, development and were used to redesign instructional activities and assess-
implementation project on-going for 20 years. While few ments. A formative evaluation was conducted during the first
researchers have the opportunity to study an instructional semester the course was offered; data included observations
program for such a long period of time, recent research con- of the instructor, classroom activities, and online discussions,
tinues to examine comprehensive design and development as well as a student survey measuring perceptions of the
projects. Below we discuss two such projects. course and their learning. The authors offer a discussion of
how these data were used to revise and improve the course.
Developing a Web 2.0 System for Community While the work by Francom et al. (2009) does not provide
and Teacher Use the same level of detail as the Cifuentes et al. (2010) study,
Research by Cifuentes, Sharp, Bulu, Benz, and Stough both are examples of comprehensive design and develop-
(2010) provides an example of a comprehensive product ment research. They report on projects in which a researcher
design study. The purpose of this 2-year study was to inves- studies design and development while comprehensive ISD
tigate “the design, development, implementation, and evalu- processes are used to produce a specific product.
ation of an informational and instructional Web site in order
to generate guidelines for instructional designers of read/
write Web environments” (p. 378). The researchers imple- Recent ID Phase Research
mented and documented the entire ISD process. Needs anal-
ysis was conducted on a practical problem—individuals with Not all DDR pertains to a comprehensive project. Instead,
disabilities and their families have difficulty gaining access some researchers examine specific phases of an ID effort.
to information about support services. Findings pointed to These studies typically relate to data gathering phases of the
the development of an online directory of resources using the ISD process (e.g., needs assessment, formative evaluation).
capabilities of Web 2.0 technologies. Design decisions were For example, Klein, Martin, Tutty, and Su (2005) identify the
based on theory including social constructivism, distributed optimal research competencies of graduate students by con-
cognition, and rapid prototyping. Formative evaluation ducting a content review of course syllabi from several lead-
occurred throughout product development. Participants ing instructional design and technology programs and by
included the design team, a variety of intended users in mul- administering a survey to faculty and students. In addition,
tiple locations, college students contributed resources to the Fischer, Savenye, and Sullivan (2002) conduct a formative
Web site, as well as internal and external evaluators. The evaluation of computer-based training on an online financial
researchers provide context-specific findings related to and purchasing system to verify the program’s effectiveness
problems and issues encountered, resources required, and and identify necessary revisions. Below, we discuss another
product impact and use. They also give generalized recom- design and development study that is representative of very
mendations for others designers of Web 2.0 solutions. recent research on a component of ID.
The Cifuentes et al. (2010) study is a good example of
comprehensive product design research. The researchers Formative Evaluation of a Learning Game
identified and analyzed a real-world problem, used theory A recent study by Sahrir (2012) is an example of DDR
and formative evaluation to inform design and development on a specific phase of ID, namely, formative evaluation.
decisions, meticulously documented these decisions, This research investigates the development of an online
144 R.C. Richey and J.D. Klein

vocabulary game for beginning Arabic language learners at a enhance the design of the PSS. In addition, elementary and
Malaysian university. The researcher employed a mixed- junior-high school teachers completed a survey about system
method approach to collect data on prototypes of the online requirements, tested the usability of two prototypes of the
learning game. Data sources included instructors, subject- PSS, kept activity logs during implementation, and partici-
matter experts, evaluators, and learners who participated in pated in post-implementation interviews. The researchers
one-to-one, small group and field test phases of formative report contextually based findings about their tool (e.g., navi-
evaluation. Characteristic of most product design studies, the gation, functionality, efficiency, and ease of learning). They
researcher provides context specific findings (i.e., the online also discuss how design and development research served as
game improved student enjoyment, immersion and knowl- a “conceptual guide to not only maintain a systematic
edge of Arabic). In addition, issues, problems and lessons approach to the development process but also to broaden the
learned are discussed to inform other designers of similar perspective of the system’s instructional implications to a
products. For example, the researcher suggests “there should holistic approach that addressed system, user, and develop-
be sessions of cooperative work and research activities ment process as a whole” (Hung et al., 2010, p. 78).
between language teachers … instructional designers and
computer experts to design and develop … effective games” Design of a Computer Support System
(p. 366). This study is particularly notable because it pro- for Multimedia Curriculum Development
vides an empirical test of how the phases of formative A study by Wang, Nieveen, and van den Akker (2007)
evaluation suggested by ISD scholars (e.g., Dick, Carey, & focuses on the design of an electronic performance support
Carey, 2009; Tessmer, 1993) can be used in actual practice. system (EPSS) to help teacher-designers in China develop
scenarios for multimedia instruction. The main purpose of
the study was “to produce a practical computer support sys-
Recent Tool Development and Use Research tem for multimedia curriculum development by following an
evolutionary prototyping approach” (p. 277). The research-
Some researchers concentrate on studying the development ers wanted to create an EPSS that was valid, practical and
and use of tools, rather than on the design of products. These effective. They created four prototypes of the tool and col-
tools may support design and development or teaching and lected data from experts and end users who completed ques-
learning processes. Many of these studies focus on computer- tionnaires and participated in focus groups. Summative
based tools and some of this research is directed toward evaluation of the tool was also conducted. During this phase
automating design and development. For example, Nieveen of the study, teacher-designers were observed using the tool
and van den Akker (1999) focus on a computer system that to create scenarios for multimedia instruction; they also pro-
serves as a performance support tool for designers during the vided suggestions for improving it. Results indicate that par-
formative evaluation phase of an ID project. In addition, ticipants found the tool to be usable and practical. An
Mooij (2002) conducted a tool study examining the develop- unintended outcome was that the tool helped “teacher-
ment and use of an instructional management system for designers become acquainted with a systematic approach to
early education. Below we describe a two other studies that multimedia instructional design” (p. 289).
are representative of recent research on tool development The Wang et al. study is particularly noteworthy because
and use. it includes summative evaluation. This type of evaluation is
often not included in DDR and is infrequently used in
Development of Performance Support practice.
Tool for Teachers
A recent comprehensive study by Hung, Smith, Harris, and
Lockard (2010) illustrates research on a tool to support the Research on Models
teaching/learning process, specifically the design and devel-
opment of a performance support system (PSS) for class- The second type of design and development research per-
room behavior management. These researchers “adopted tains to studies of the development, validation and use of
design and development research methodology … to system- models. These studies focus on the design and development
atically investigate the process of applying instructional models and processes themselves, rather than their demon-
design principles, human–computer interaction, and soft- stration. While it is possible to conduct model research in
ware engineering to a performance support system” (p. 61). conjunction with the development of a product or program,
The study was conducted in six phases that mirrors an ISD many model studies concentrate on previously developed
approach. Qualitative and quantitative techniques were instruction, and consequently are not project-specific. Model
used to collect data from several sources. For example, a research may address the validity or effectiveness of an
Delphi technique was used with subject matter experts to existing or newly constructed development model, process
12 Design and Development Research 145

or technique. In addition, these studies often seek to identify of selecting visual images, the varying functions of pictures,
and describe the conditions that facilitate successful design symbols and signs, as well as a range of distortions that can
and development. Since model research studies are oriented occur during message transmission.
toward a broad analysis of design and development pro- Reviews of large bodies of research and theory are likely
cesses, they tend to be more generalizable than product and to cover many settings, be they instructional, transfer, or
tool studies. design and development. Therefore, this technique facilitates
Model research tends to address three major related the identification of factors that are not context-specific or
phases—model development, model validation, and model learner-specific. This is the case with the Voss model.
use. Here we review very recent representative model
research in each category. Identifying the Components of a Transfer Model
Like many researchers and practitioners interested in improv-
ing workplace performance, Hillsman and Kuptritz (2010)
Recent Model Development Research seek to identify empirically based predictors of transfer.
Specifically, they focus on elements in the physical work
Model development research may result in new, enhanced, or environment. Their work is an example of a study that can be
updated models that guide the ID process or a part of the viewed as content-specific research, but then ID researchers
process. Such research has produced a rapid prototyping ID can also interpret their findings in terms of design and devel-
model (Jones & Richey, 2000), components of a model of ID opment. More specifically, the research can be viewed as
competencies (Vallachia, Marker, & Taylor, 2010), and a model development since it provides justification for includ-
Web-based knowledge management system model that pro- ing an entire class of variables into a design model directed
vides for its continuing development (Plass & Salisbusry, towards transfer of training.
2002). Model research encompasses a wide range of settings The Hillsman and Kuptritz study was a multi-methods
and participants and it employs a variety of research method- project (both qualitative and quantitative) that collected data
ologies (see Richey & Klein, 2007). from 50 supervisors who had participated in 4 hours of inter-
We examine two recent model development studies that personal communication training and then applied their new
address very different design problems using research meth- skills working with their employees on the job for 6 months.
ods that are totally different from each other. However, both The research involved conducting 6 hours of field observa-
contribute to the advancement of design and development tions, surveys, and structured personal interviews. In addi-
models. tion, there was an archival review of work records.
The results show that workplace design did “contribute to
A Model for the Design of Visual Information transfer outcomes. Supportive as well as unsupportive work-
Message design is a specialized task of those who select place design features were elicited as most often facilitating
and develop instructional materials, and it is an area or impeding transfer” (Hillsman & Kuptritz, 2010, p. 23).
informed by a broad knowledge base. Consequently, Voss While this research clearly has implications for workplace
(2008) conducted an extensive literature review resulting in design, it also adds to the body of literature that seeks to
the development of a model to guide designers as they work model those factors that impact transfer of training. Thus, it
with one particular type of message—two-dimensional also informs training designers of those aspects of context
visual images that will transfer information to the learner/ that are critical to the success of their interventions. These
viewer. This study explores the research literature of mes- researchers do not fully develop a design-related model, but
sage design, cognitive psychology, neurology, and infor- instead they identify the building blocks required to con-
mation theory to identify those principles that govern visual struct a comprehensive model.
communication, mental imagery, and visual memory. The
literature and the resulting model suggest that visuals have
their own set of rules that are based upon the nature of per- Recent Model Validation Research
ception rather than the view of communication as being
controlled by language. While the ID literature is rife with models of the design pro-
While reviewing the literature is an important early step cess, far less attention is paid to the validation of these mod-
in conducting any research, a literature review is not typi- els. Such validation is an empirical process that demonstrates
cally used as a research methodology in the IDT field. the effectiveness of a model’s use in a real-world setting (i.e.,
However, large-scale reviews such as Voss’s provide an external validation) or provides support for the various com-
opportunity to build an empirically based model that covers ponents of a model (i.e., internal validation) (Richey, 2005).
many variables. For instance, the Voss model addresses pre- In some validation research, experienced design practitioners
attentive and attentive brain functions, the mental processes are used as subject matter experts to authenticate a design
146 R.C. Richey and J.D. Klein

model or specific design phases. For example, Cowell (2001) MI Model designers filled out a model usability survey detail-
interviews current designers to substantiate the regular use of ing their reactions to the model. Product impact, on the other
needs assessment techniques (even though other terms are hand, was tested by using the two design team’s products.
often used for the process). In other research, learner data Five sessions with eight to ten learners each were conducted
confirms the model. (See Roszkowski & Soven, 2010, for using the MI-oriented workshop, and another five similar ses-
their research which validates an updated Kirkpatrick sions were conducted using the ISD product. Posttest and
evaluation model.) We summarize two other recent DDR attitude-toward-training data were collected. While both
studies which highlight the characteristics of model valida- groups felt confident in their new skills, participants who
tion research. were trained with the MI materials scored slightly (but
significantly) higher on the posttest and learning seemed to be
Updating and Validating Gilbert’s Behavioral stimulated by the use of the MI instructional strategies.
Engineering Model Tracey’s research supports the use of the MI Design
Thomas Gilbert’s Behavioral Engineering Model (BEM) has model. Like other design and development studies, it
profoundly influenced designers who work in employee exemplifies comprehensive model validation techniques per-
training environments, and it has been credited with the orig- formed under real-world design conditions.
ination of cause analysis. (See Gilbert, 1978, for a full dis-
cussion of this model.) Crossman’s (2010) research examines
BEM’s relevance in the contemporary workplace. Recent Model Use Research
The participants in Crossman’s study are 600 fire fighters
and the specific area of interest is safety culture, the motiva- While it is not unusual for model validation research, such as
tion to follow safety rules. The fire fighters completed a sur- Tracey’s (2009) study, to address usability issues, there is
vey whose items reflected the environmental elements of the another genre of design and development research that
BEM—information (i.e., communication), resources, and emphasizes how models are used. Many of these studies
incentives. Data were analyzed using correlations and path focus on the conditions that impact model use; these show
analyses. Crossman found that the combined effects of the the interplay between varying design and development con-
three variable categories did influence safety motivation. texts and model effectiveness. For example, Roytek (2000)
Furthermore, she found that incentives directly impacted conducted a comprehensive case study which focuses on two
safety motivation while absorbing the indirect effects of design projects using rapid prototyping procedures; this
communication of information and resources. The environ- study is designed to determine which contextual factors,
mental facet of Gilbert’s long standing model was validated strategies, and events facilitate or impede project success.
in this setting. Other research focuses on the designers themselves to under-
Crossman’s study exemplifies an internal model valida- stand exactly how the design and development process is
tion asking whether the parts of the model are justified. It is actually used. Visscher-Voerman and Gustafson (2004) con-
an empirical study that relates to an actual work environ- ducted interviews with designers working in diverse settings
ment. It is statistically sound and based in both theory and and reviewed related project materials to determine the pro-
practice. However, there are other ways to approach model cedures designers used and their rationales for these
validation. approaches. Recent model usability research continues in a
similar vein with much of the current work concentrating on
Testing the Impact of the Multiple Intelligence the role of technology.
Design Model
Tracey (2009) uses very different tactics to validate her ID The Rapid Implementation of e-Learning
model which blends multiple intelligence (MI) theory with Many academic programs are faced with the prospect of
traditional instructional systems design. The study has two changing quickly from face-to-face delivery of their courses
parts—a designer usability test and an examination of prod- to on-line learning. Coetzee and Smart (2012) present a case
uct impact. As such, it provides both internal and external study describing the process of developing a module and
validation of the MI Design Model. placing it on learning management system (LMS). In doing
Designer usability was tested by randomly assigning two so, they demonstrate the merger of two models—the tradi-
Masters-trained designers to a 2 hours team building work- tional ADDIE (analysis, design, development, implementa-
shop project using the MI Design Model; two similar design- tion, evaluation) design model and The Technology Process
ers were assigned to the same project using the Dick and model used in technology development projects. The situa-
Carey ISD Model. (See Dick et al., 2009 for a full discussion tion in this study was realistic in that the university instructor
of this model.) Work conditions were the same for each design was working essentially alone with only one other person
team. Following completion of the workshop materials, the giving advice. Resources were limited. There was little lead
12 Design and Development Research 147

time and thus the modules were used as developed. The level thinking. Contextual factors (e.g., accessibility) had the
course thus moved from a face-to-face delivery to a blended most impact on technology use.
delivery. Subsequent units were modified based upon student Model researchers typically look forward to positive
feedback. This case study demonstrates how two recognized results that confirm their model’s utility. Hart’s (2008)
models can be modified to meet the demands of a given research, on the other hand, produces less satisfying results.
instructional situation. The models can be scaled up or scaled The data, however, provide an empirical basis for changing
down to meet the varying needs of a particular intervention. models to accommodate the real-world conditions.
The Coetzee and Smart study is particularly useful
because model use is not examined in a technology-rich
environment. This demonstration takes place in university Summary of Key Characteristics
located in an underdeveloped country. Nonetheless, the two
models are successfully adapted to the conditions present in We have described 11 studies published since 2007 which
their specific context. are representative of the most recent design and development
research. These studies encompass the various types of both
Teacher Technology Integration product and tool research and model research. They were
There are many models directed towards classroom teachers conducted in a variety of work and geographical settings and
as they integrate technology into their lessons. The existing address diverse problems currently being faced by the field.
research, however, provides little data supporting teacher use The researchers also use a wide assortment of approaches
of either classic ID principles or the consistent use of tech- and methodologies to study design and development prob-
nology in their classes. Hart (2008) uses a “think aloud” pro- lems. Table 12.1 below summarizes this recent research.
tocol to study how middle school language arts and social
studies teachers actually integrate technology into their les-
son plans While these teachers were not applying specifically Source of Design and Development Problems
designated design or technology integration models, Hart
explores which model components are used by identifying Design and development research typically stems from prob-
the design decisions made and the rationale for these deci- lems encountered in the workplace (Richey & Klein, 2007).
sions. Eight teachers (four of whom had graduate education Of the 11 studies reviewed in this chapter, seven of them are
in instructional design) completed a background survey, then directly rooted in real-life problems. For example, this
a technology design task using the think-aloud techniques, research was used to answer questions such as: What facili-
and a post-design interview. Hart found that in general these tates transfer of training to the job? How can we help teach-
teachers demonstrated a reliance on mental planning rather ers to take on the role of designers? How can we get vital
than use of design principles. Technology was not incorpo- information to individuals and families?
rated into all teachers’ lessons in a meaningful way (even In keeping with the increased use of technology in educa-
though that was the assigned task). Moreover, when it was tion and training, it is not unusual for design and develop-
used, the technology did not tend to support student higher ment studies to have a technology focus. Over half of the

Table 12.1 An overview of recent representative design and development research

Problem Source Setting Methodology Evaluation
Work-place/ Adult Ed. Higher Quali- Quanti-
Study society Technology Theory and Trng. P-12 Educ. tative tative Survey Formative Summative
Product and tool
Cifuentes et al. (2010) X X X X X
Francom et al. (2009) X X X X X X X
Hung et al. (2010) X X X X X X X X X
Sahrir (2012) X X X X X
Wang et al. (2007) X X X X X X X X X
Model
Coetzee and Smart (2012) X X X X X X
Crossman (2010) X X X
Hart (2008) X X X X
Hillsman and Kuptritz (2010) X X X X X
Tracey (2009) X X X X
Voss (2008) X X
148 R.C. Richey and J.D. Klein

studies we highlighted had a technology emphasis, and five conducted in a higher education setting (Coetzee & Smart,
of the seven studies with a workplace focus also examined 2012; Francom et al., 2009; Sahrir, 2012). In all three cases,
problems related to technology. These studies concentrated participants included faculty who were responsible for
on online learning, Web site design, technology integration, designing instruction for their students.
electronic performance support systems, and gaming. All of
these topics reflect emerging technologies as well as current
trends in the IDT field. Research Methodology
Over half of this body of recent research also reflects
theoretical problems and issues. While some of these studies The majority of design and development studies use multi-
(such as Voss’s, 2008 exploration of factors affecting the method approaches typically blending both qualitative and
perception of visual messages) emanate only from an inter- quantitative methods (Richey & Klein, 2007). This may be a
est in theory, others (such as Francom et al.’s, 2009 study reflection of the complexities of most projects and the multiple
of online course design using Merrill’s First Principles) sources of the problems address in such research. However,
combine a theoretical orientation with practical concerns. qualitative methods were dominant. Nine of the 11 studies
Most DDR addresses problems which have multiple reviewed in this chapter employed qualitative techniques. We
sources. In our sample of 11 recent studies only the model believe that this is a typical phenomenon. The qualitative meth-
research with a theoretical focus seemed to have a more sin- ods, however, vary widely. They include the use of case studies,
gular focus. This conclusion, however, may only be a pecu- participant interviews, focus groups, field observations, activity
liarity of the particular sample of studies we reviewed. logs, archival reviews, and think-aloud techniques.
Many studies also employ quantitative methods and may
at times use experimental designs. Not surprisingly, evalua-
Research Settings and Participants tion phases of design and development research often rely
upon assessment measures. Probably the most common
Design and development research problems (like ID itself) quantitative method involved the use of surveys and ques-
are typically contained in a specific context which includes tionnaires. For example, Crossman (2010) used survey data
distinct participants. ID is now used extensively in business collected from fire fighters to validate the Gilbert model and
and industrial settings, healthcare organizations, community Hung et al. (2010) surveyed classroom teachers to identify
and government agencies, as well as schools and universi- the requirements of their performance support system.
ties. The 11 studies described in this chapter reflect this Standard statistical techniques such as correlations and path
diversity for the most part. Four of the five product and tool analyses were then employed.
studies are situated in educational settings—two at the P-12 Finally, there is a critical methodological issue somewhat
level and two in higher education. On the other hand, half of unique to design and development research. In many of
the model research reviewed pertains to employee training. these studies, the researcher also serves as the designer/
All but one of the recent design and development studies developer. This situation is a common and often unavoidable
reviewed in this chapter were conducted in a setting that by-product of the practical constraints of studying real-life
included adults as participants, although in some cases the design projects. These conditions occur in all of the recent
participants were adult learners rather than instructional product and tool studies summarized in this chapter and in
designers alone. For example, product and tool research was one of the model studies (i.e., Coetzee & Smart, 2012). In
done in the context of a learning community that included these cases, data validity can be an issue, but when special
the parents of children with disabilities and county extension attention is given to instrument design, data collection and
agents (Cifuentes et al., 2010). Model research was con- triangulating multiple sources of data, the concerns have
ducted with managers in a training setting (Hillsman & been addressed. The position of the designer/researcher is
Kuptritz, 2010) and with fire fighters employed by a local comparable to the role of participant observer in qualitative
government (Crossman, 2010). research, and similar data collection tactics are employed.
Even recent design and development research conducted
in P-12 school and higher education settings focuses primar-
ily on adults. Our review identified three studies that concen- The Role of Evaluation
trated on school teachers. The product and tool studies by
Hung et al. (2010) and Wang et al. (2007) examined the Evaluation is a major part of the design and development pro-
design and use performance support tools for teachers while cess and correspondingly plays a role in DDR although it is
the model use study by Hart (2008) focused on how teachers far more prominent in product and tool research than in model
integrate technology into their lesson plans. Furthermore, research. Since designers who follow a systems approach
3 of the 11 studies we reviewed in this chapter were typically evaluate the intervention during development,
12 Design and Development Research 149

researchers who study the design and development of a prod- Coetzee, J. S., & Smart, A. (2012). Rapid implementation of e-learning
uct or tool often collect similar evaluation data to determine using a technology design model. In N. A. Alias & S. Hashim (Eds.),
Instructional technology research, design, and development:
its impact on learning. As expected, the comprehensive Lessons from the field (pp. 219–237). Hershey, PA: IGI Global.
research projects such as those conducted by Cifuentes et al. Cowell, D. (2001). Needs assessment activities and techniques of
(2010), Hung et al. (2010) and Francom et al. (2009) include instructional designers: A qualitative study (Doctoral dissertation,
formative evaluation tasks in their studies. However, all of Wayne State University, 2000). Dissertation Abstracts International
A, 61(910), 3873.
the product and tool studies and one of the model studies we *Crossman, D. C. (2010). Gilbert’s Behavioral Engineering Model:
summarized in this chapter included some form of formative Contemporary support for an established theory. Performance
evaluation. Sahrir (2012) placed a major emphasis on this Improvement Quarterly, 23(1), 31–52.
process when he empirically tested how the various phases Definition and Terminology Committee of the Association for
Educational Communications & Technology. (2007). Definition. In
of formative evaluation can be used by university instruc- A. Januszewski & M. Molenda (Eds.), Educational technology: A
tors. Typically these data include learner assessments, but it definition with commentary (pp. 1–14). New York, NY: Routledge.
often includes designer reactions as well. Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of
Of special consideration is the study by Wang et al. (2007) instruction (7th ed.). Upper Saddle River, NJ: Merrill.
Driscoll, M. P. (1984). Paradigms for research in instructional systems.
which also includes a summative evaluation to investigate Journal of Instructional Development, 7(4), 2–5.
the impact of a performance support tool on teachers who Fischer, K. M., Savenye, W. C., & Sullivan, H. J. (2002). Formative
develop curriculum. The inclusion of both formative and evaluation of computer-based training for a university financial sys-
summative evaluation data is an encouraging trend in the tem. Performance Improvement Quarterly, 15(1), 11–24.
Francom, G., Bybee, D., Wolfersberger, M., & Merrill, M. D. (2009).
IDT literature and we hope it continues. Biology 100: A task-centered, peer-interactive redesign. TechTrends,
Researchers who study design models are less likely to 53(4), 35–42.
concentrate on evaluation data unless they are studying eval- Gilbert, T. (1978). Human competence: Engineering worthy perfor-
uation models. However, some researchers such as Coetzee mance. New York, NY: McGraw-Hill.
Hart, S. M. (2008). The design decisions of teachers during technology
and Smart (2012) may include formative evaluation in their integration. (Unpublished doctoral dissertation). Wayne State
studies of model use. Additionally, others develop and imple- University, Detroit, MI.
ment an intervention to test the efficacy of the model. In this Hillsman, T. L., & Kuptritz, V. W. (2010). Another look at the relative impact
process learner assessment data is often used. of workplace design on training transfer for supervisory communication
skills. Performance Improvement Quarterly, 23(3), 107–130.
*Hung, W-C., Smith, T. J., Harris, M. S., & Lockard, J. (2010).
Development research of a teachers’ educational performance sup-
Conclusions port system: The practices of design, development, and evaluation.
Educational Technology Research and Development, 58(1), 61–80.
Jones, T. S., & Richey, R. C. (2000). Rapid prototyping in action: A
In the past, instructional design strategies were supported developmental study. Educational Technology Research and
primarily by research on the learning process. While that is Development, 48(2), 63–80.
still a valuable source of information, ID is now substanti- Klein, J. D. (1997). ETR&D—Development: An analysis of content
ated by a much broader array of research. One trend in the and survey of future direction. Educational Technology Research
and Development, 45(3), 57–62.
field is the use of design and development research. It estab- Klein, J. D., Martin, F., Tutty, J., & Su, Y. (2005). Teaching research to
lishes practical and theoretically sound solutions to the many instructional design & technology students. Educational Technology,
problems faced in the IDT field. While this type of research 45(4), 29–33.
is not yet commonplace, it is growing. The studies reviewed Merrill, M. D. (2002). First principles of instruction. Educational
Technology Research and Development, 50(3), 43–59.
in this chapter reflect this phenomenon. Design and develop- Mooij, T. (2002). Designing a digital instructional management system
ment research is being conducted in many parts of the world. to optimize early education. Educational Technology Research and
It is being applied to many new topics and areas of concern. Development, 50(4), 11–23.
These researchers are providing the field not only with inno- Nieveen, N., & van den Akker, J. (1999). Exploring the potential of a
computer tool for instructional developers. Educational Technology
vative examples of how such studies are conducted, but with Research and Development, 47(3), 77–98.
new knowledge about how to design and develop interven- Plass, J. L., & Salisbusry, M. W. (2002). A living-systems design model
tions which address critical problems in education, training, for web-based knowledge management systems. Educational
and organizational improvement. Technology Research and Development, 50(1), 35–57.
Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A meth-
odology for creating and improving design theories. In C. M.
Reigeluth (Ed.), Instructional design theories and models, volume
References II: A new paradigm of instructional theory (pp. 633–651). Mahwah,
NJ: Lawrence Erlbaum Associates, Publishers.
*Cifuentes, L., Sharp, A., Bulu, S., Benz, M., & Stough, L. M. (2010). Reiser, R. A. (2012). What field did you say you were in? Defining and
Developing a Web 2.0-based system with user-authored content for naming our field. In R. A. Reiser & J. V. Dempsey (Eds.), Trends
community use and teacher education. Educational Technology and issues in instructional design and technology (3rd ed., pp. 1–7).
Research and Development, 58(4), 377–398. Upper Saddle River, NJ: Merrill.
150 R.C. Richey and J.D. Klein

Richey, R. C. (1997). Research on instructional development. Educational higher learning experience. In N. Alias & S. Hashim (Eds.),
Technology Research and Development, 45(3), 91–100. Instructional technology research, design and development: Lessons
*Richey, R. C. (2005). Validating instructional design and development from the field (pp. 357–368). Hershey, PA: IGI Global.
models. In J. M. Spector & D.A. Wiley (Eds.), Innovations in instruc- Sullivan, H., Ice, K., & Niedermeyer, F. (2000). Long-term instruc-
tional technology: Essays in honor of M. David Merrill (pp. 171– tional development: A 20-year ID and implementation project.
185). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Educational Technology Research and Development, 48(4),
Richey, R. C., & Klein, J. D. (2005). Developmental research methods: 87–99.
Creating knowledge from instructional design and development Tessmer, M. (1993). Planning and conducting formative evaluation:
practice. Journal of Computing in Higher Education, 16(2), 23–38. Improving the quality of education and training. London: Kogan Page.
*Richey, R. C., & Klein, J. D. (2007). Design and development research: Tracey, M. S. (2009). Design and development research: A model vali-
Methods, strategies and issues. Mahwah, NJ: Lawrence Erlbaum dation case. Educational Technology Research and Development,
Associates, Publishers. 57(4), 553–571.
Richey, R. C., & Klein, J. D. (2008). Research on design and develop- Vallachia, S. W., Marker, A., & Taylor, K. (2010). But what do they
ment. In J. M. Spector, M. D. Merrill, J. van Merrienboer, & M. P. really expect? Employer perceptions of the skills of entry-level
Driscoll (Eds.), Handbook of research for educational communica- instructional designers. Performance Improvement Quarterly, 22(4),
tions and technology (3rd ed., pp. 748–757). Mahwah, NJ: Lawrence 33–51.
Erlbaum Associates, Publishers. *van den Akker, J. (1999). Principles and methods of development
*Richey, R. C., Klein, J. D., & Nelson, W. (2004). Developmental research. In J. van den Akker, R. M. Branch, K. Gustafson, N.
research: Studies of instructional design and development. In D. Nieveen & T. Plomp (Eds.), Design approaches and tools in educa-
Jonassen (Ed.), Handbook of research for educational communica- tion and training (pp. 1–14). Dordrecht, The Netherlands: Kluwer
tions and technology (2nd ed., pp. 1099–1130). Mahwah, NJ: Academic Publishers.
Lawrence Erlbaum Associates, Publishers. Visscher-Voerman, I., & Gustafson, K. L. (2004). Paradigms in the
Richey, R. C., Klein, J. D., & Tracey, M. W. (2011). The instructional theory and practice of education and training design. Educational
design knowledge base: Theory, research and practice. New York, Technology Research and Development, 52(2), 69–89.
NY: Routledge. Visser, L., Plomp, T., Armiault, R. J., & Kuiper, W. (2002). Motivating
Richey, R. C., & Nelson, W. (1996). Developmental research. In D. Jonassen students at a distance: The case of an international audience.
(Ed.), Handbook of research for educational communications and tech- Educational Technology Research and Development, 50(2),
nology (pp. 1213–1245). New York, NY: Simon & Schuster. 94–110.
Roszkowski, M. J., & Soven, M. (2010). Did you learning something *Voss, D. R. (2008). The development of a model for non-verbal factors
useful today? An analysis of how perceived utility relates to per- impacting the design of visual information. Unpublished doctoral
ceived learning and their predictiveness of satisfaction with training. dissertation, Wayne State University, Detroit, MI.
Performance Improvement Quarterly, 23(2), 71–91. Wang, F., & Hannafin, M. J. (2005). Design-based research and tech-
Roytek, M. A. (2000). Contextual factors affecting the use of rapid nology-enhanced learning environments. Educational Technology
prototyping within the design and development of instruction Research and Development, 53(4), 5–23.
(Doctoral dissertation, Wayne State University, 1999). Dissertation Wang, Q., Nieveen, N., & van den Akker, J. (2007). Designing a com-
Abstracts International A, 61(01), 76. puter support system for multimedia curriculum development in
Sahrir, M. (2012). Formative evaluation of an Arabic online vocabulary Shanghai. Educational Technology Research and Development,
learning game prototype: Lessons from a Malaysian institute of 55(3), 275–295.