Special Report

A Review of Innovative Teaching

Methods
D1X XRebecca T. Sivarajah, D2X XMD, D3X XNicole E. Curci, D4X XMD, D5X XElizabeth M. Johnson, D6X XMD, D7X XDiana L. Lam, D8X XMD,
D9X XJames T. Lee, D10X XMD, D1X XMichael L. Richardson, D12X XMD

Teaching is one of the important roles of an academic radiologist. Therefore, it is important that radiologists are taught how to effectively edu-
cate and, in turn, to act as role models of these skills to trainees. This is reinforced by the Liaison Committee on Medical Education which has
the requirement that all residents who interact with and teach medical students must undergo training in effective methods of teaching. Radiol-
ogists are likely familiar with the traditional didactic lecture-type teaching format. However, there are many newer innovative teaching methods
that could be added to the radiologist’s teaching repertoire, which could be used to enhance the traditional lecture format. The Association of
University Radiologists Radiology Research Alliance Task Force on Noninterpretive Skills therefore presents a review of several innovative
teaching methods, which include the use of audience response technology, long-distance teaching, the flipped classroom, and active learning.
© 2018 Published by Elsevier Inc. on behalf of The Association of University Radiologists.

INTRODUCTION interact with and teach medical students to undergo training

in effective methods of teaching and evaluation (1).

A
s radiologists, we are adept at interpreting medical
Didactic lectures are a well-known form of teaching. How-
images. However, how did we learn this informa-
ever, there are many newer innovative teaching skills, pedagogi-
tion? Some of this knowledge came from the count-
cal techniques, and forms of educational technology that could
less hours spent studying radiology books and journals.
also be added to a radiologist’s teaching repertoire. These could
However, much of our understanding also came from being
be used to enhance lectures radiologists are already giving or be
taught by others. Thinking back to the great teachers you
used in addition to the traditional lecture format.
had—what made them stand out? What made them effective
The Association of University Radiologists Radiology
teachers? What interesting techniques did they use to teach?
Research Alliance Task Force on Noninterpretive Skills
This review is relevant because as a radiologist you will find
therefore presents a review of innovative teaching methods.
you are often called on to teach. Radiologists, especially in the
Our goal is to present a practical compendium of useful tools
academic setting, are tasked with the education of fellows, resi-
and techniques that allow education to become more engag-
dents, and medical students. However, all radiologists, including
ing, more interactive, and more effective:
those in private practice, find themselves in some teaching
role—teaching fellow radiologists, other nonradiology physi-
cians, technologists, and patients. Therefore, it is important that Part I: Audience response technology
radiologists are taught how to effectively educate and, in turn, Part II: Long-distance teaching
to act as role models of these skills to trainees. Part III: Flipped classroom
This is reinforced by the Liaison Committee on Medical Part IV: Active learning
Education, whose Standard 9.1 requires all residents who
PART I
Acad Radiol 2019; 26:101 113 Audience Response Systems
From the Department of Radiology, Penn State Health Milton S. Hershey Med-
ical Center, Hershey, Pennsylvania (R.T.S.); Department of Radiology, Univer- An audience response system (ARS) is used by presenters to
sity of Michigan, Ann Arbor, Michigan (N.E.C.); Department of Radiology, solicit information from their audience. Many educational
University of South Florida, Tampa, Florida (E.M.J.); Department of Radiology,
University of Washington, 4245 Roosevelt Way NE, Seattle, WA 98105 (D.L.L., benefits have been ascribed to ARS, such as anonymity and
M.L.R.); Department of Radiology, University of Kentucky, Lexington, Ken- increases in interactivity, participation, engagement, and
tucky (J.T.L.). Received January 28, 2018; revised February 28, 2018;
accepted March 24, 2018. The authors are members of the Association of
measurable learning performance (2).
University Radiologists Radiology Research Alliance Task Force on Noninter-
pretive Skills. This special review is a white paper created by that task force.
This research did not receive any specific grant from funding agencies in the Background or History
public, commercial, or not-for-profit sectors. Address correspondence to:
M.L.R. e-mail: mrich@uw.edu The first ARSs were developed in the 1950s by the United
© 2018 Published by Elsevier Inc. on behalf of The Association of University States Air Force and were used to quiz personnel with

101
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

multiple choice questions (MCQs), which were embedded to match our ARS technology. Rather than continuing to
into training films (3). In the 1960s, ARSs were built and force instructors and students to lie in the Procrustean bed of
installed in lecture halls at Stanford University (1966) and MCQ and true or false questions, there are a number of
Cornell University (1968) (4,5). These early custom systems promising and more modern pedagogical approaches that
were hard-wired into a classroom and were somewhat costly have been developed.
($17,000 $150,000) (6).
By the 2000s, ARS technology had evolved into more
21st Century Pedagogical Techniques
affordable systems ($1000 $2000) consisting of portable
response devices (“clickers”) paired with a USB receiver
1. Bidirectional ARS
attached to a computer system (7). Since 2007, smartphones and
other portable computing devices have become increasingly
common among students and instructors, making it possible to ARSs have traditionally operated in only one direction,
use these as web-based ARS input and receiving devices (8). that is, the presenter polls the audience. Modern ARSs allow
Any attempt to publish a comprehensive feature compari- bidirectional interrogation, so that the audience can anony-
son chart of the rapidly growing list of available ARS tools mously ask the presenter for additional information or clarifi-
quickly becomes outdated. However, Awan et al. (9) recently cation. In our experience, this two-way flow of questions has
published just such a chart comparing the three most com- been quite popular. It should be no surprise that an audience
monly used ARSs: RSNA Diagnosis Live (Radiological Soci- that is shy about answering questions in front of their peers
ety of North America [RSNA], Oak Brook, IL), Poll will also be shy about asking a potentially “dumb” question
Everywhere (Poll Everywhere, San Francisco, CA), and in front of their peers.
NearPod (Nearpod, Aventura, FL).
Each different ARSs will usually have its own set of fea- 2. Asking questions on the fly
tures and its own unique interface. Each system will pose its
own learning curve for both presenters and audiences. Some It can take a considerable amount of time to create MCQs in
systems require user accounts, which allow student responses advance of a lecture, especially if one also has to enter them into
to be stored and individual performance to be tracked. Other an ARS. In contrast, if one sticks to questions, such as “What is
systems give the presenter the ability to divide the audience the diagnosis?” or “What is the finding?” or “What is the differ-
into teams and lead them in various teaching games. Some ential?,” there is no need for this sort of tedious prelecture prep-
systems may require the presenter to download an application aration. It is amazingly liberating to go into a case conference
or plug-in to provide integration with Microsoft PowerPoint knowing that one’s ARS will allow one to pose naturally occur-
(Microsoft, Redmond, WA). Other systems require special ring questions on the fly and react immediately to questions
response devices (clickers) and USB receivers. However, from one’s residents. For those of us who love teaching by the
most modern ARSs allow an audience to submit their Socratic method (13), on-the-fly questions in a free-text format
responses via the web, either using a specific smartphone are the heart and soul of the process.
application or web browser. Web-based systems have the
advantage of being able to separately poll an audience regard- 3. Peer instruction
less of the presentation software used.
Peer instruction (PI) is a technique developed by Eric
Mazur for teaching physics students (14) and exploits student
Pedagogy, Not Technology
interactions to enhance understanding of difficult concepts.
The feature sets of current ARSs are growing constantly. This can be especially helpful when the ARS poll shows sub-
However, it can be easy to get caught up in an arms race of stantial disagreement among students. In this situation, stu-
features and technological gimmicks. In contrast to the vast dents are invited to briefly discuss the question with their
improvements in ARS technology, ARS techniques have neighbors and are then repolled before the correct answer is
remained relatively static over the past 50 years—the technol- revealed. Several studies of physics and biology students have
ogy has evolved, but the way it is used has not. shown that this sort of peer discussion enhances understand-
As a pedagogical tool, a depressingly large fraction of ARS ing, even when none of the students in a discussion group
presentations in radiology still consist of peppering the audience originally knows the correct answer (12,15 17).
with true or false question and MCQs. These same types of PI has also shown benefits when dealing with classes in
questions have been used since 1926, when the multiple choice which the students have a widely different level of back-
SAT was introduced as a faster way of testing college applicants ground knowledge about a topic. In general, students with
(10,11). We may be using computers to do the work, but we more background knowledge benefit more during PI than
continue to poll our audiences as if it were still 1926. students with less background knowledge. However, students
There is, therefore, a growing sentiment that it is not the with less background knowledge gain as much during PI as
technology but the pedagogy that matters the most (8,12). It students with more background knowledge gain during tra-
is about time that we upgraded our ARS teaching techniques ditional instruction (16).

102
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

4. Radiology-specific ARS techniques documenting these evaluations very easy (8). One can end
each of one’s teaching presentations with a few ARS ques-
Traditional case conference tions that ask the students to evaluate the presentation. Many
A large part of radiology practice involves going from the commercial ARSs will save these evaluations in a session log
findings to the most likely diagnosis. Traditional case confer- for each presentation.
ences are used to train this skill. Rather than focusing on one Addressing different learning styles
resident at a time, one can poll multiple residents simulta- Also, radiologists and radiology residents exhibit a variety
neously. With a web-based ARS, one can poll students at of learning styles. Different styles of learning require different
more than one site. styles of teaching, such as the three leading theories of human
“Inverted” case presentation learning: behaviorism, cognitivism, and constructivism (20).
Another large part of radiology practice involves going If we adapt our teaching to match these different styles of
from a known diagnosis to generating a list of expected find- learning, it follows that we should also use ARS differently
ings. This skill can be strengthened by “inverting” the case with each of these styles (8).
conference. Instead of showing an unknown case and asking Future technology
for a diagnosis, one can present a diagnosis and then ask one’s For the most part, ARS technology has transitioned to web-
residents to list the pertinent radiographic findings as free text or cloud-based technology. Vendors no longer depend on audi-
via the ARS. ence to have “clickers” or the presenter to have a dedicated
Progressive refinement case presentation receiver. Future ARS may involve monitoring the audience’s
When residents do not know an answer, one can itera- nonverbal response to a presentation. These future systems may
tively add bits of information that progressively steer them take advantage of wearable technology monitoring heart rate or
closer and closer to the answer while repolling them with the blood pressure. Cameras on personal computing devices may
ARS. As one adds progressively more and more facts, it is fas- also be used to follow pupillary dilatation and eye gaze direction.
cinating to watch the list of diagnoses quickly converge to Additionally, the definition of “audience” will continue to
the right answer. For example, a first response from one’s res- evolve. Social media and live streaming technology continue to
idents might mention “calcifications” within a bone lesion. A challenge our traditional understanding of audience response.
follow-up question about matrix types might then elicit Today’s technology allows an affordable voting system—
“chondroid matrix.” Further hints could lead to a differential an electronic, group version of the Socratic method—that
of chondroid bone lesions, a search for other findings, and an facilitates interactive teaching and enhances lecturing in med-
eventual diagnosis of multiple hereditary exostosis syndrome icine. In our experience, this modality improves the quality
with degeneration into a chondrosarcoma. of clinical learning and deserves further evaluation (13). ARS
Optimal imaging workup provides an excellent tool for improving interactive learning
In the course of a case conference, one can poll one’s stu- in radiology residents. However, it is not the technology but
dents as to the optimal imaging protocols for computed the pedagogy that matters the most (12). It is long past time
tomography or magnetic resonance imaging. One can also to upgrade our ARS teaching techniques to match our ARS
poll the audience as to the optimal imaging workup for a par- technology. It is time to move on to more innovative ARS
ticular disease. It is easy to parlay questions such as this into an techniques that are well adapted to radiology and its different
interactive teaching session built around the American Col- styles of learning.
lege of Radiology Appropriateness Criteria (18).
Differential diagnosis
Real-life cases often lack a single correct answer. It is the PART II
radiologist’s job to therefore pose an appropriate differential
Long-distance Teaching
diagnosis in one’s report. A properly designed ARS can allow
students to enter multiple items for the differential and pres- Long-distance teaching can be accomplished by means of
ent all of these responses in alphabetical order. It is fascinating many conventional Internet tools, such as email, text messag-
to see how often the aggregate differential diagnosis of a ing, and web forums. There are also a variety of online
group of residents approaches that of an expert radiologist. courseware platforms such as WizIQ (WizIQ, Gurgaon,
India), Blackboard (Blackboard, Washington, DC), and
5. Real-time evaluation of teaching sessions by students Moodle (Moodle HQ, Perth, Australia). However, for pur-
poses of this review, we will focus on remote conferencing
Many radiology faculty currently find themselves on some systems, which allow real-time streaming of online meetings
variant of a clinician-educator pathway for promotion. A with audio, video, and screen sharing.
major tool for clinician-educator promotion is the assembly Daily or weekly didactic conferences are a major component
of a portfolio documenting one’s teaching activities (19). Stu- of most radiology residency programs in the United States. For
dent evaluations of one’s teaching should be a key element of programs with residents rotating at multiple sites, or for affiliated
such a portfolio, although few faculty consistently collect stu- programs that combine their didactic curriculum, traveling con-
dent evaluations. Appropriate use of an ARS can make straints and clinical responsibilities can complicate conference

103
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

scheduling and physical attendance. Remote meeting technol- easier for all voices to be heard. A powered speaker with a
ogy has become increasingly popular, allowing speakers and built-in amplifier can make it much easier for all listeners in
audiences at multiple geographic locations to participate in con- the room to hear those speaking online. To enable adequate
ference. Teleconferencing or videoconferencing from multiple screen sharing in a physical conference room setting, a large
sites not only increases residents’ exposure to expert faculty TV screen or a computer projector is key.
teaching but also saves residents’ time and expense from com- With a large system of interlinked microphones and powered
muting between sites for conference (21). Shah et al. reported speakers, a bit of audio feedback is inevitable. This can usually
many positive outcomes following implementation of a remote be alleviated by careful placement of one’s microphones with
conferencing program (22). Time savings achieved by the resi- respect to one’s speakers, as well as adjustment of the volume
dents resulted in increased efficiency and reduced workload for controls. One can also greatly minimize audio feedback by hav-
faculty members, as eliminating commute time saved the ing each member of the conference voluntarily mute themselves
remote participants up to 60 minutes. Residents who are post- when they are not speaking. Another annoying source of audio
call or unable to leave clinical areas can participate, as well as feedback occurs when two people in the same room connect to
those attending professional meetings out of town. a conference on two different devices. A final source of audio
There are some disadvantages of remote conferencing. For feedback comes from a feature offered by many conferencing
example, many programs experience decreased participation of systems: allowing audience members to connect by telephone.
remote residents in the conference, which limits interactive dis- Some conferencing systems allow little or no control or muting
cussion. Previous studies investigating the use of remote confer- of the audio of those connecting via telephone. In a very few
encing described a sense of exclusion felt by remote participants cases, this has led to cancelled conferences due to badly behaving
(23). In addition, technological issues such as poor audio trans- telephone audio. For this reason, we no longer enable telephone
mission or loss of image quality can be problematic (21). call-ins to our conferences, and our audio feedback issues have
The number of web-based video conferencing services been greatly reduced.
(VCS) available at a given time waxes and wanes as start-ups
rise and fall and as new companies are taken over and
The Pedagogy of Online Conferencing
absorbed by the older ones. Any comprehensive listing of
such services will therefore be somewhat obsolete by the Most of the web conferencing services offer the same basic
time it reaches print. A 2017 review by PC Magazine posted functionality. Given that, it is not so much the technology
a detailed feature-comparison table of 10 different VCSs pop- but once again the pedagogy that matters the most (8,12).
ular at the time of publication (24). We will therefore now focus on techniques for optimizing
Web-based tools work on all major platforms (Mac OS X, education with an online conferencing system. In a decade of
Windows, Linux, iOS, and Android). Although many of these experience with online conferencing, the authors have
tools have developed stand-alone apps, they can be used with no amassed a long list of technical and social tips for running a
software other than a web browser. These browser-based services successful online conference (25).
include ClickMeeting (Gda
nsk, Poland), Zoho Meeting (Pleas-
anton, CA), GoToMeeting (LogMeIn, Boston, MA), Adobe Increasing Interactivity
Connect (Adobe, San Jose, CA), Skype (Microsoft, Redmond, While one can use video conferencing systems to give tradi-
WA), WebEx (Cisco, San Jose, CA), and others (24). Provided tional didactic lectures, we prefer to spend our valuable contact
features differ among tools; some allowing users to speak to each time with our students with presentations that are much more
other through their computers’ microphones, whereas others interactive. Therefore, most of our online conferences are given
teleconference through a telephone bridge. Prices vary depend- in the unknown case conference format, using some type of
ing on the number of desired participants. For example, GoTo- ARS. Many VCSs include an online chat function that allows
Meeting prices start at $29/month for up to 50 people or $199/ viewers to send messages to other viewers or to the entire group.
month for up to 500 people. It runs on most computer and These features were found to be extremely useful in a weekly,
smartphone platforms. Collaboration tools include screen sharing multi-institutional case conference (25). In this conference,
with mouse and keyboard controls, multiple presenters, text attendees take turns presenting interesting cases from their own
chat, mobile access, and video conferencing. Additional features practices. The usual presentation format involves the presenta-
such as polling and survey capability, postmeeting metrics, and tion of an unknown case, with findings and differential diagnosis
recording options are available (23,25,26). elicited from the other attendees. The answer, if known, is then
presented, followed by a short slide-based discussion of the rele-
vant diagnostic and treatment aspects of the case, plus a few
Hardware Considerations
salient references. The VCS control panel allows the moderator
Regardless of the computer platform used, the sound quality to switch among presenters and to control other aspects of the
can be greatly improved by the addition of a high-quality conference.
microphone. For single user sites, a headphone with built-in The main disadvantage of a built-in VCS chat function is
microphone is often ideal. For multiple users in a physical that it does not anonymize the attendees. For many of our
conference room, an omnidirectional microphone makes it resident case conference, we therefore use ARSenic, a simple

104
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

online ARS that anonymizes the users and their responses (8). Our Friday conference has turned out to be a great “one
With ARSenic and other online ARSs that allow free-text size fits all” conference for medical students, residents, fel-
responses, one can use the ARS in a bidirectional fashion, lows, and attending radiologists. This type of conference is an
allowing users not only to give anonymous answers but also especially good training tool for our residents and fellows.
to ask anonymous questions to the presenter. This bidirec- Many of our cases are presented by fellows, who will end
tionality has been very popular with our residents. their year of training with a collection of dozens of cases fully
If there are multiple users who wish to speak at the same worked up in slide form.
time, audio collisions are inevitable. These collisions can be min- This project has turned out to be an enjoyable conference
imized with the appropriate use of some of the features built with regulars who attend zealously. As an index of its popularity,
into the user interface of many VCSs. In one of our online case we held the conference last year on the days right after Thanks-
conferences, those wishing to present cases state their intention giving and Christmas. As academicians, we are bemused that
in the chat window. The moderator then transfers the screen after spending years struggling to get trainees to come to confer-
sharing to each of these presenters when it is their turn. This ences, we have finally stumbled into a conference format that
flow of control can also be enhanced using the tiny video win- people want to attend, even on their days off.
dows shown for each user in the user interface of many VCSs. We suspect that this conference has also some value as a
When one wishes to make a statement or ask a question, one recruiting tool, not only for academic positions but also with
can signal this by raising one’s hand. the private practitioners who regularly attend our conference.
We often schedule visiting applicants for fellowship and fac-
ulty members on Fridays so that they can attend our confer-
Social Aspects of Online Conferencing ence. We hope that potential applicants viewing our
There is great potential for strong social aspects to online confer- conference will come away with a favorable idea of what it
encing. This initially took us somewhat by surprise. In retrospect, would be like to work at our institutions.
this is not so surprising. One of the major reasons for attending We value the chance to pick the brains of our colleagues at
any medical meeting is to spend time with one’s friends and col- other institutions. We often show cases for which we have
leagues. One of us has hosted an online musculoskeletal (MSK) no answers and ask for their assistance. We always hope that
case conference for over 9 years. We quickly found that meeting someone else will be able to tell us the right answer or suggest
once a week was a lot more fun than meeting three or four times ideas for further workup. However, even when no one
a year. This simple enjoyment is probably one of the major rea- knows the answer, it is very comforting to know that one is
sons for the growth and enduring success of that conference. among such good company in ignorance.
We also greatly value seeing the wide range of pathology Our conference allows us to enrich our experience with
from other institutions and the ability to confer with colleagues disorders not seen locally. For example, one of our sites has a
on difficult cases (Fig 1). Our conference has also spawned sev- large experience with MSK tumors but little exposure to
eral collaborative educational projects, such as an online journal trauma. Other sites see huge trauma volumes but have few
club, a published book of conference cases, peer-reviewed case tumors. Our current mixture of sites makes for a rich mix of
reports, and an online MSK hardware atlas. most areas of MSK practice.

Figure 1. Sites that have joined one of our weekly online conferences as of January 2018. (Color version of figure is available online.)

105
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

Occasionally, cases coming up in our conference are har- processes with interactive application of knowledge through
bingers of new disorders or treatment modalities. For exam- problem solving, case-based scenarios, or interactive discus-
ple, a case of aseptic lymphocyte-dominated vasculitis- sions—all under the guidance of the teacher (34).
associated lesion presented at our conference was the first The basic elements of a flipped classroom are as described
time many of us had heard of this phenomenon. The condi- previously. However, experienced educators have developed
tion, an intense lymphocytic response to a certain type of the four pillars to describe flipped learning, a more specific
metal-on-metal hip prosthesis, is now the reason for a large, form of the flipped classroom. To implement flipped learn-
nationwide recall of this particular implant (27). ing, educators must integrate the following additional com-
Several private practitioners regularly attend our confer- ponents into their practice utilizing the acronym F-L-I-P: a
ence. In some cases, they are the sole person with MSK train- Flexible teaching environment, a shift in Learning culture,
ing in their group and are delighted to spend an hour per Intentional content, and a Professional educator (35,36).
week with friends who actually care about and enjoy discus-
sing Mazabraud syndrome (28,29). They also value the aca- Flexible environment: Flipped classrooms allow a vari-
demic input from 10 to 20 other MSK specialists reading the ety of learning modes including group work, independent
literature and reporting on new and interesting concepts. study, research, performance, and evaluation.
They also find it helpful to see how orthopedics is practiced Shift in Learning culture: In the flipped learning model,
in different parts of the country. there is a deliberate shift from a teacher-centered classroom to
a student-centered approach, where in-class time is meant for
Unexpected Spin-offs From Video Conferencing exploring topics in greater depth and creating richer learning
Although our conference was originally envisioned as a case opportunities. Students become the center of learning.
conference, several related educational activities have subse- Intentional content: Flipped educators evaluate what
quently evolved over time. content they need to teach directly and they continually
Early on in the course of our conference, we decided that think about how they can use the flipped learning model
it would be helpful to periodically gather the best cases from to help students gain conceptual understanding.
our weekly conference into some permanently accessible for- Professional educators: In the flipped learning model,
mat. We decided to do this in the form of an electronic professional educators are vital. During class time, they
book. In exchange for providing case write-ups to the book, observe students, provide relevant feedback, and decide
our contributors gain authorial lines on their curricula vitae. how to maximize the face-to-face time between teachers
Our first book of 50 cases, Intergalactic Online MSK Conference: and students.
Cool Case Book 1 (30), was published in May 2011 and is cur-
Proponents of Flipped Learning Believe That It Enhances
rently available on Amazon’s Kindle store, Apple’s online
the Learning Experience—but Why?
bookstore, and Barnes & Noble Nook store. It has been
joined subsequently by three other volumes of cases (31 33). With traditional learning, when students are doing home-
A number of the cases shown in our conference have suffi- work or applying what they learned in class, it is usually at
cient appeal and teaching points that we encourage their home. If they have a question, a teacher is not available to
authors to write them up for a formal peer-reviewed case help. In comparison, in a flipped learning environment,
report in journals such as Radiology Case Reports (https:// when students are applying new knowledge and concepts,
www.journals.elsevier.com/radiology-case-reports/). Gath- the teacher or professional educator is present to answer ques-
ering similar cases from multiple institutions for case reviews tions because this work is being done in the classroom.
and other projects has been another natural outgrowth from In a flipped classroom, students are doing lower levels of
our online conference. work (gaining knowledge and comprehension) outside of
class and focusing on higher forms of cognitive work (appli-
cation, analysis, synthesis, or evaluation) in class where they
PART III have the support of peers and the instructor.
In a flipped classroom, teachers are therefore able to spend
Flipped Classrooms
more of their face-to-face time supporting students in deeper
What Is a Flipped Classroom? learning processes such as knowledge application, assimilation
In a traditional classroom or learning environment, learning is of information, and problem solving instead of just being
centered on a didactic teaching method. A teacher stands at content disseminators. A teacher’s interaction with students
the front of the classroom and delivers a lecture, and then in a flipped classroom can be more personalized with more
assignments and problem-solving exercises are performed at individual support (36,37).
home. In a flipped classroom, the order of events is “flipped.” A second advantage of a flipped classroom is that students
Educational material and instructional content is delivered can review lecture materials at their own pace and review
before the class session via online lectures, videos, or readings and replay parts of the lecture as many times as they need. It
which the student can access and review at their own pace. also allows students who may have missed class to still view
The class session is then dedicated to more active learning the lecture content (38).

106
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

Does a Flipped Classroom Model Work? preclass content delivery, student centered in class learning activ-
ities, and then assessment of student learning (46).
The flipped classroom approach has been shown to be effec-
Creating preclass content: The goal is to provide key
tive in secondary schools. In 2011, educators in Michigan’s
information to the students before class that can then be rein-
Clintondale High School flipped every classroom. In one
forced with in class activities. This could take the form of
social studies class, two classes with identical material and
recorded video lectures or textbook or journal readings. An
assignments were run: one flipped and one conventional.
easy way for a first-time “flipper” to develop video content
The flipped class had many students who had already failed
may be to record the lectures they are already giving and
the class. After 20 weeks, students in the flipped classroom
archive these lectures. Recording a lecture could be as simple
were outperforming students in the traditional classrooms.
as using a recorded PowerPoint narration using PowerPoint’s
No students in the flipped classroom scored lower than a C+.
built-in narration. There are also many screening casting soft-
In the previous semester, 13% had failed. The traditional
ware packages available to help with this. Specific to radiol-
classroom showed no change (36,39).
ogy education, there are a variety of e-learning modules that
The flipped learning model has been shown to be effective
could supplement these lectures including RadCasts, RadTo-
in the postgraduate and medical educational community as
rials, and RadGames.
well. At Stanford, the biochemistry curriculum was rede-
When designing and delivering preclass content, it is
signed following the flipped education model, with class time
important to provide clear learning objectives for each class
used for interactive discussions of clinical vignettes and out-
session so students know how to prepare. It is important to
of-class time used to view online presentations made by the
focus on key foundational concepts that students must under-
instructors. Positive course reviews increased and attendance
stand so that they can they can then practice applying these
increased from 30% to 80% (40).
concepts during class. Efficiency is important and experts in
Use of flipped classroom during delivery of a renal phar-
flipping have found that shorter lectures (10 15 minutes) are
macotherpy module improved student’s performance com-
more effective than longer lectures (34,43).
pared to performance of students the previous year that
Design in-class activities: The purpose of the in-class
underwent the same module in a more traditional setting
activities is for the student to take the information taught in
(34,41).
the preclass assignment and engage in activities that reinforce
In 2016, The University of Vermont Robert Larner Col-
and apply these concepts. It is important to avoid just rele-
lege of Medicine became the first US medical school to com-
cturing content that was already provided in the preclass
mit to flipping its entire required curriculum (42).
assignment. Sometimes, a 1- to 3-minute micro-lecture is
When a flipped classroom approach was used in a medical
helpful to reinforce learning or address an apparent gap in
school radiology elective, the flipped classroom learners
knowledge identified when engaging in the in-class activity.
achieved higher pretest to posttest improvement compared to
The in-class activities should promote higher orders of think-
other teaching strategies, and students enjoyed the components
ing and application. Many of the in-class activities incorpo-
of the blended learning sections over traditional learners (43).
rate active learning and some examples of active learning
Another study tested the effects of flipped learning in a
exercises that could be used are as follows (47):
radiology medical clerkship to explore whether it would
better facilitate fact learning and development of clinical
Individual Activities
decision-making skills (44). Subjects who participated in the
flipped learning cohorts had greater interest in learning,
 Individual exercises: These provide an opportunity for the
increased enjoyment, and higher task value than the tradi-
students to practice skills. This could include labeling,
tional didactic instruction cohorts. In addition, the change
rank ordering, multiple choice, problem solving or true or
in posttest vs pretest scores was 10.5% higher in the flipped
false, and completion.
learning group than in the didactic instruction group.
 Role play: Students are given a situation and a role to play
In a Mayo Clinic Internal Medicine Residency Program, a
of a character in the situation.
study was conducted to determine whether the flipped class-
 Questioning strategies: Questions which the students will
room approach would improve the quality improvement
be asked are planned by the instructor before the lecture.
knowledge of the residents. They found that the residents
Each question is written out and is related to a learning
who participated in the flipped classroom approach demon-
objective. In case of no response or incorrect responses,
strated improved quality improvement knowledge compared
the instructor will also be prepared to ask easier or lower
to the control group (45).
learning level questions, which will lead the student to
answer the original question.
 Personal vignette: Given a topic or learning objective, the
How Does One Begin to Implement Flipped Learning Into
students are asked to relate it to their real experiences (per-
Their Classroom or Learning Environment?
sonal or professional) by telling a brief story about it.
Experts in flipped learning stress that there is not just one way to  Progress quizzes: Short self-tests that are not graded. Answers
“flip.” There are three components that need to be developed: are provided to the students after completing the quiz.

107
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

Group Activities tool across many learning environments and may have a place
in radiology education.
 Buzz/Brainstorm sessions: A small group of students work
within a determined time limit to answer a question or
solve a problem and come to a conclusion. PART IV
 Group work exercise: Students are given a problem or sit-
Active Learning
uation to solve in a 5 10 minute period of time.
 Problems: A group of students works on given problems Active learning is an approach to instruction in which stu-
within a specified time. The instructor discusses the correct dents engage the material they study through reading, writ-
answers at the end of the activity. ing, talking, listening, and reflecting. Active learning is
 Complete case studies: These are real-world descriptions basically “anything course-related that all students in a class
of problems with all accompanying data. Groups are asked session are called upon to do other than simply watching, lis-
to resolve the problem within a given period of time. tening and taking notes” (https://www.cte.cornell.edu/teach
Each group makes recommendations while the instructor ing-ideas/engaging-students/active-learning.html).
acts as moderator. We all can remember those great lectures that we
 Diagnostic sessions: Groups diagnose a problem, situation, attended—ones where the hour passed without us realizing
process, etc. and we were actively engaged throughout the lesson. Unfor-
 Pyramiding (Snowball groups): Given a problem, students tunately, we can also all remember lectures where we strug-
first work alone, then in pairs, and finally in foursomes gled to keep our eyes open, often leaving the lectures with
(maximum) and compare, refine, and revise their conclu- “sleepy notes” or finding ourselves distracted. The problem
sions and recommendations. with the traditional model for education is that it has been
primarily teacher-centered, with the student as a passive
Activities to Promote Class Discussion learner as information is provided in a 50- to 60-minute lec-
ture without engagement of the students.
 Controlled discussion: This is used after a lecture. Students
However, studies have shown that this is not the optimal
ask questions and make comments while the instructor
way for students to learn (48). Instead, an active learning
controls the process which goes on for a prescribed period
model that is more student-centered and requires active par-
of time.
ticipation of each student has been shown to be most effec-
 Choices: Given data on an event and several choices, stu-
tive. The two main goals of active learning (46) are to
dents are asked to discuss all choices. They may select a
choice, justify it, and give the consequences. 1. shift focus from the teacher to the learner
 Debate: This is an organized and civil argument moder- 2. promote higher order cognitive tasks through active
ated by the instructor. engagement with course content
 Discussion of handouts: These are special documents,
reports, pictures, etc., that are discussed.
 Simulation: This presents cases, problems, scenarios, etc., Advantages of Active Learning
in which the students must role play. A critical situation is
In a review of the literature on active learning, Prince found
discussed and analyzed, and decisions are made about how
broad support for the core elements of active learning (49),
to resolve the situation.
 Read and discuss: This is a short reading followed by a dis- with improvements in both short- and long-term learning
compared to the standard didactic lecture. Some advantages
cussion.
of active learning include the following:

Assessment of Student Learning

Improved student attention: It has been shown that
The teacher can use a variety of methods to provide feedback after an initial settling-in period of 5 minutes, students
about student learning. These could include a preclass or pre- readily assimilate material in a traditional lecture for the
activity learning assessment (which helps the teacher under- next 5 minutes. Ten to 20 minutes into the lecture, how-
stand which concepts from the preclass assignment may need ever, assimilation falls off rapidly. Active learning uses skills
additional review), real-time assessment during the in-class that help break up the lecture and keep students interested
learning, and graded assignments or projects that test the abil- and focused.
ity of the student to not only memorize information but also Higher order learning skills used: Active learning is
analyze, synthesize, and evaluate concepts. also better at integrating higher level learning skills such
In summary, the flipped classroom approach is a novel way as applying, analyzing, and evaluating compared to pas-
to teach students, allowing the valuable teacher-student sive listening which utilizes lower end skills of remember-
interaction time to be used to review higher order educa- ing and understanding. This allows learners to develop
tional activities instead of just used as a time to lecture and better problem-solving skills and improves retention of
disseminate information. It has been shown to be an effective information.

108
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

More motivating: Active learning can be more moti- this time up, the lecturer can pause every 15 minutes or so,
vating and interesting to learners as it often requires them taking a break for 2 3 minutes. During this time, the teacher
to apply knowledge learned to clinically relevant situa- can allow students to compare their class notes so far to other
tions. It also allows more frequent and immediate feed- students. This not only keeps the attention of students but
back to students. allows learners to clarify their own understanding and com-
Helps perfect additional professional skills: Active pare their perspectives to their partner (54).
learning also provides opportunities to learn and practice One minute papers: In this technique, in the last 10 15
skills for professional work such as team skills, communi- minutes of class the lecturer asks the following questions,
cation, negotiation, and peer assessment of performance “What is the most important thing that you have learned
(46). today? Write one to two questions that you have regarding
the lecture. What subject would you like to know more
Research on Active Learning about?” Have students write down answers.
Student summary of another’s answer: Have one stu-
In a review of research, Prince examined the evidence for the
dent (ie, lower level resident) answer a question about the
effectiveness of active learning and found broad but uneven
lecture and have another student (ie, higher level resident)
support for all forms of active learning (49). A meta-analysis
summarize that answer and then add or elaborate.
of 225 studies comparing traditional lecture to active learning
Stage setting: Give questions at the beginning of class and
in university STEM (Science, Technology, Engineering, and
instruct them to listen for answers within the lecture.
Math) courses found that active learning reduces failure rates
Recall material: At the beginning of class, have students
from 32% to 21% and increases student performance on
take 3 5 minutes to write down the most important con-
course assessments and concept inventories by 0.47 standard
cepts from a previous class lecture or reading assignment.
deviations (48). In another study, the impact of two types of
Active review: After the instructor summarizes the class,
active learning, team-based learning and case-based group
ask students to spend 2 3 minutes reading through notes
discussion, was evaluated in a medical school pathology
to identify points of confusion.
course. The study found that students in the lower academic
Muddiest point: Instructor has students respond to the
quartile showed better examination performance after using a
following question: “A question I still have about this topic
team-based approach (50,51).
but have been afraid to ask is . . . (ie muddiest point).”
Have them write the questions down and then turn in
How to Perform Active Learning (47,54).
Relevance: Students retain more information when it
Active learning is achieved when students interact with the
seems more clinically relevant. In medical education, a per-
educational content in any way that can promote active
sonal clinical scenario or case could be discussed. The lec-
thought which encourages them to think about the material
ture could even include a live patient.
rather than just transcribe the content (52). There are many
active learning techniques that can be used.
Active Group-based Techniques
More active or complex tasks can also be used during lecture
Interactive Lectures time which require more group-based activities. Examples of
In its simplest form, active learning can be integrated into these activities include think-pair-share and jigsaw.
already designed lectures. Therefore, it can even be utilized Think-pair-share: A thought-provoking question is posed
in large lecture room settings. Activities can be done during by the teacher to the class. Students think of an answer and write
the lecture that help students actively think about the material it down (think). Students then form pairs and discuss responses
being taught and recognize areas with which they may need (pair). Students then share answers with a team, group, or entire
more help. These techniques are brief activities, which can class (share) (Fig 2). This is believed to enhance student prob-
be used to break up a traditional lecture helping to engage lem-solving skills and enhance their ability to see errors or
students throughout the entire lecture time. Some examples insights into the thinking of others (46).
include the following: It may be helpful to present a brief example of the think-
pair-share technique applied to radiology. One abundant
The testing effect: Give a test at the beginning or end of source of case material for this sort of exercise is an online
the class. Students who took tests (even if they did not count journal such as Radiology Case Reports (https://www.journals.
toward their overall grade) were found to have better long- elsevier.com/radiology-case-reports/). For example, one
term retention of material (53). To implement this, an ARS might give the students the history of an otherwise 56-year-
could be used. To further increase the active learning, a old healthy woman presenting with atraumatic bilateral ankle
teacher could have the students come up with the test ques- pain and bilateral Achilles tendon lesions, as shown in Figure 3
tions themselves, based on the lecture material. (55). After pondering this information, students write down
Pause and clarify/update notes: Student attention span their thoughts and then pair up with another student. Each
starts to dwindle about 20 minutes into a lecture. To break pair will likely decide that the patient has lesions in both

109
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

Figure 2. Think-pair-share technique.

(Color version of figure is available
online.)

Figure 3. T1-weighted magnetic reso-

nance images demonstrate small foci of
increased signal in the medial-posterior
margins of both Achilles tendons, con-
sistent with small partial width tears.

Achilles tendons, but will have differing opinions as to the complication of the fluoroquinolone antibiotic she was given
cause of these lesions. When the pairs meet as a group, these for her pharyngitis.
different opinions will merge into a collective differential Jigsaw: In a jigsaw, a topic is divided into separate but related
diagnosis list, including items such as normal variant, tumor, pieces and requires the learners to become experts in a subject
infection, or trauma. The group will also be struck by the area and then teach that topic to peers who have become
bilateral nature of the findings and the lack of any significant experts in their topics. The class is divided into teams and each
trauma. At this point they will come up with a list of further member of the team is assigned one piece. She or he then works
questions they need to answer. Some students will then com- to understand and become an expert on her or his piece. Then,
pare the findings in the images to those of various tendon dis- each person teaches the other team members about what he or
eases in a radiology text. Others will do a PubMed search for she knows about that piece—thereby putting together the jig-
causes of bilateral tendinopathy. Yet, others will search the saw puzzle (Fig 4). This could also be used in a slightly different
patient’s history for further clues. Eventually, the group may way—where the class is divided into groups and each group is
learn of the phenomenon of drug-induced tendinopathy and given a piece. The groups work together to understand their
conclude that this patient’s bilateral tendon tears are a piece. Then, the class is redivided into mixed groups, with one

110
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

Figure 4. Jigsaw technique. (Color ver-

sion of figure is available online.)

member from each group in each new group. Each person in Application exercise phase (usually about 75% of the
the group teaches the rest of the group what he or she knows— class time): In this phase, students demonstrate their ability
again forming the puzzle. to apply the knowledge they have just prelearned by working
in groups to complete an assignment. All groups complete the
Team-based Learning same assignment or exercise which usually involves determin-
Team-based learning (TBL) is an evidence-based collaborative ing the best solution to a given significant, relevant problem.
learning teaching strategy designed around units of instruction, The assignments should promote collaboration among stu-
known as “modules,” that are taught in a three-step cycle: prep- dents in each group and use the phase 1 and 2 knowledge. At
aration, in-class readiness assurance testing, and application- the end, teams then reveal their collaborative group answers
focused exercise (http://www.teambasedlearning.org). to the entire class, and the teacher facilitates a classroom dis-
Before implementing TBL, classes must be divided into cussion between teams to explore the topic.
groups or teams (usually five to seven members). Every effort Team-based learning implementation is based on four
should be made to ensure the teams are diverse with the goal underlying principles:
of equally distributing intellectual talent among the groups. 1. Groups should be properly formed and are fixed through
Groups are fixed throughout the entire course and are ideally the whole course.
determined by the teacher (not student selected). 2. Students are accountable for their prelearning and for
Preclass preparation: During the preclass preparation, working in teams
learners independently study material that has been preas- 3. Team assignments must promote both learning and team
signed by their teacher (ie, reviewing a chapter in a book, development
reading articles, viewing PowerPoint slide presentations, or 4. Students must receive frequent and immediate feedback
viewing prerecorded lectures). The goal of the preclass prepa-
ration is to review fundamental concepts that will help the
Games
learner solve problems during the in class activity. Learning
objectives should be clearly defined by the teacher. An education game is defined as “an instructional method
Readiness assurance phase: At the beginning of class, stu- requiring the learner to participate in a competitive activity with
dents take an individual readiness assurance test, which is preset rules (56).” Educational games have been used in medical
designed to assess their individual preparedness for class (usually education as a way to review specific medical topics and try to
5 20 MCQs). They then take the same test with their groups, make the learning process more active and fun (57,58).
called the team RAT. However, groups must determine a con- Medical jeopardy is an example of a quiz-type educational
sensus answer for each question. The answers to the questions game in which students are presented with general knowledge
are revealed immediately to the group using a special Feedback clues in the form of answers and they must then phrase their
Assessment Technique card (groups scratch off an answer and a responses in the form of questions. A PowerPoint version of this
star is revealed if the answer is correct). Both tests usually count game allows educators to create their own jeopardy style game
toward the student’s grade. The cards are collected and the using a free downloadable template (59). One study compared
instructor then focuses a short mini-lecture to review concepts medical student’s performance, satisfaction, and retention of
that are still confusing to the students. knowledge on a jeopardy game format teaching method vs a

111
SIVARAJAH ET AL Academic Radiology, Vol 26, No 1, January 2019

didactic lecture format (60). They found that both groups long distance education were also discussed. Through an
showed a significant improvement in their knowledge on an example in MSK radiology, it was shown how an online
immediate posttest compared to the pretest. However, on a conference could promote increased interactivity between
posttest performed 2 months after, retention of knowledge was radiologists of all levels at many difference institutions. The
significantly better in the game format group. flipped classroom approach is a novel way to teach students,
Breast Imaging Bingo is another example of an educational allowing the valuable teacher-student interaction time to be
game that has been developed to supplement traditional didactic used to review higher order educational activities. It has
lectures (61). The game incorporates popular music to reinforce been shown to be an effective tool across many learning
memory retention, with lyrics designed to be similar to the cor- environments, including radiology education. The final
rect answer or principle that the question assessed. A session of innovative teaching method to be reviewed was active
this game was played at the 2015 American Osteopathic College learning, which engages students and improves their atten-
of Radiology Breast Imaging Innovation Conference. Partici- tion through using engaging techniques such as working in
pants found the game enjoyable and felt they would use this for- groups and reflecting during the lecture. We encourage
mat to supplement their conference or lectures. radiology educators to incorporate innovative teaching
A wide variety of medical board and card games have also methods, such as the four described in this paper, into their
been developed to aid in teaching medical concepts. These pedagogy as they have been shown to enhance the learning
games range from the common “2 stage” quiz games where a process.
question has to be answered, the answer is evaluated in some
way, and the next question follows to more complex 4 stage
games which also requires abstraction of the issue and then plan- REFERENCES
ning the next step. A detailed review of medical games is pro-
1. Liaison Committee on Medical Education. Functions and structure of a
vided in an article by Bochennek et al. (62). medical school. Standards for accreditation of medical education pro-
Medical puzzles such as crosswords and anagrams are addi- grams leading to the MD degree. 2017. Available at: http://lcme.org/pub-
tional educational games that can be used to help reinforce med- lications/#Standards. Accessed May 4, 2018.
2. Kay RH, LeSage A. Examining the benefits and challenges of using audi-
ical concepts. These puzzles can be done individually or in ence response systems: a review of the literature. Comput Educ 2009;
groups to incorporate small group learning. In one study, cross- 53:819–827.
word puzzles were given to small groups of undergraduate med- 3. Judson E, Sawada D. Audience response systems in higher education:
applications in cases chap. 2. In: Audience response systems: insipid
ical students. Students reported that the crossword puzzles were contrivances or inspiring tools?, Hershey, PA: Information Science Pub-
beneficial in helping them identify key concepts, contributed to lishing; 2006:26–39.
their learning, and engaged them in collaborative practice (63). 4. Abrahamson AL. Audience response systems in higher education: appli-
cations and cases chap. 1. In: A brief history of networked classrooms:
In summary, incorporating active learning into education has effects, cases, pedagogy, and implications, Hershey, PA: Information
been shown to be beneficial. Active learning has been shown to Science Publishing; 2006:1–25.
engage students and improve their attention. It has also been 5. Littauer R. Instructional implications of a low-cost electronic student
response system. Educ Technol 1972; 12:69–71.
shown to improve retention of information by developing 6. Shapiro JA. Electronic student response found feasible in large science
higher order critical thinking skills, and it can also foster com- lecture hall: inexpensive, homemade system sparks student attention
munication and cooperative learning skills. Active learning can and participation. J Coll Sci Teach 1997; 26:408–412.
7. Hatch J, Jensen M, Moore R. Manna from heaven or “clickers” from hell.
take many forms. As described previously, active learning can J Coll Sci Teach 2005; 34:36–39.
range in complexity from adding a few simple active learning 8. Richardson ML. Audience response techniques for 21st century radiology
techniques to an already established lecture to development of a education. Acad Radiol 2014; 21:834–841. doi:10.1016/j.acra.2013.09.026.
9. Awan OA, Shaikh F, Kalbfleisch B, et al. RSNA diagnosis live: a novel
complete team-based learning curriculum. Active learning can web-based audience response tool to promote evidence-based learning.
also incorporate fun elements such as games. When developing Radiographics 2017; 37:1111–1118. doi:10.1148/rg.2017160107.
a teaching curriculum for radiology residents or medical stu- 10. Mathews J. Just whose idea was all this testing? Fueled by technology,
nation’s attempt to create a level playing field has had a rocky history
dents, active learning strategies should be kept in mind. Washington Post. A.6.
11. Ravitch D. The fall of the standard-bearers. Chron High Educ 2006;
52:844.
CONCLUSION 12. Mazur E. Farewell, lecture? Science 2009; 323:50–51.
13. Brezis M, Cohen R. Interactive learning in medicine: Socrates in elec-
We have reviewed four innovative teaching methods that tronic clothes. QJM 2004; 97:47–51.
could be implemented into a radiologist’s teaching practice. 14. Mazur E. Peer instruction: a user’s manual. New York City, NY: Prentice
Hall, 1997.
These educational skills go beyond the routine didactic slide 15. Fagen AP, Crouch CH, Mazur E. Peer instruction: results from a range of
lecture format and help actively engage students in learning. classrooms. Phys Teacher 2002; 40:206–209.
ARSs were described and examples of how they could be 16. Lasry N, Mazur E, Watkins J. Peer instruction: from Harvard to the two-
year college. Am J Phys 2008; 76:1066–1069.
used during a lecture to encourage interactive learning were 17. Smith MK, Wood WB, Adams WK, et al. Why peer discussion improves
reviewed. ARSs now allow students to anonymously ask student performance on in-class concept questions. Science 2009;
questions to the lecturer in real time and allow teachers to 323:122–124. doi:10.1126/science.1165919.
18. American College of Radiology Expert Panels. ACR appropriateness cri-
ask “on the fly” open-ended questions as most systems allow teria. 2017. Available at: https://www.acr.org/Quality-Safety/Appropri-
free-text answers from students. Techniques for optimizing ateness-Criteria. Accessed May 4, 2018.

112
Academic Radiology, Vol 26, No 1, January 2019 INNOVATIVE TEACHING METHODS

19. Lewis P. Promotion resources: teaching portfolio template. 2017. Available 42. Krisberg K. Flipped classrooms: scrapping lectures in favor of active
at: http://www.aur.org/Secondary-Alliances.aspx?id=514. Accessed May learning AAMC News. 2017.
4, 2018. 43. Belfi LM, Bartolotta RJ, Giambrone AE, et al. “Flipping” the introductory
20. Williamson KB, Gunderman RB, Cohen MD, et al. Learning theory in radiol- clerkship in radiology: impact on medical student performance and per-
ogy education. Radiology 2004; 233:15–18. doi:10.1148/radiol.2331040198. ceptions. Acad Radiol 2015; 22:794–801. doi:10.1016/j.acra.2014.11.003.
21. Rapaport H, Loomis J, Kagetsu NJ, et al. Megaconference: a radical 44. O’Connor EE, Fried J, McNulty N, et al. Flipping radiology education right
approach to radiology resident education with full-day weekly conferen- side up. Acad Radiol 2016; 23:810–822. doi:10.1016/j.acra.2016.02.011.
ces. J Am Coll Radiol 2013; 10:51–56. doi:10.1016/j.jacr.2012.08.011. 45. Bonnes SL, Ratelle JT, Halvorsen AJ, et al. Flipping the quality improve-
22. Shah CC, Deloney LA, Donepudi C, et al. Establishment of a simple and ment classroom in residency education. Acad Med 2017; 92:101–107.
inexpensive remote radiology conference. J Am Coll Radiol 2009; 6:884– doi:10.1097/ACM.0000000000001412.
885. doi:10.1016/j.jacr.2009.07.001. 46. Fornari A, Poznanski A. How-to guide for active learning International
23. Lackey AE, Pandey T, Moshiri M, et al. Productivity, part 2: cloud stor- Association of Medical Science Educators. 2015.
age, remote meeting tools, screencasting, speech recognition software, 47. University of Pittsburgh’s University Center for Teaching & Learning.
password managers, and online data backup. J Am Coll Radiol 2014; Designing in-class activities: Examples of active learning activities. 2017.
11:580–588. doi:10.1016/j.jacr.2013.11.021. Available at: http://www.cidde.pitt.edu/wp-content/uploads/2014/04/
24. Martinez J, McLaughlin MK. The best video conferencing software of Designing_In-Class_Activities-Handout-Examples_Of_Active_Learnin-
2017 PC Magazine. 2017. g_Activities.pdf. Accessed May 4, 2018.
25. Richardson ML, Petscavage JM, Hunter JC, et al. Running an online 48. Freeman S, Eddy SL, McDonough M, et al. Active learning increases stu-
radiology teaching conference: why it’s a great idea and how to do it dent performance in science, engineering, and mathematics. Proc Natl
successfully. Acad Radiol 2012; 19:746–751. doi:10.1016/j. Acad Sci U S A 2014; 111:8410–8415.
acra.2011.10.030. 49. Prince M. Does active learning work? A review of the research. J Eng
26. Bhargava P, Dhand S, Lackey AE, et al. Radiology education 2.0—on the Educ 2004; 93:223–231.
cusp of change: part 2. ebooks; file sharing and synchronization tools; web- 50. Koles PG, Stolfi A, Borges NJ, et al. The impact of team-based learning
sites/teaching files; reference management tools and note taking applica- on medical students’ academic performance. Acad Med 2010; 85:1739–
tions. Acad Radiol 2013; 20:373–381. doi:10.1016/j.acra.2012.11.001. 1745. doi:10.1097/ACM.0b013e3181f52bed.
27. Watters TS, Cardona DM, Menon KS, et al. Aseptic lymphocyte-domi- 51. Koles P, Nelson S, Stolfi A, et al. Active learning in a year 2 pathology
nated vasculitis-associated lesion: a clinicopathologic review of an curriculum. Med Educ 2005; 39:1045–1055.
underrecognized cause of prosthetic failure. Am J Clin Pathol 2010; 52. Gifkins J. What is “Active Learning” and why is it important?. 2015. Avail-
134:886–893. doi:10.1309/AJCPLTNEUAH8XI4W. able at: http://www.e-ir.info/2015/10/08/what-is-active-learning-and-
28. Case DB, Chapman Jr CN, Freeman JK, et al. Best cases from the AFIP: why-is-it-important/. Accessed May 4, 2018.
atypical presentation of polyostotic fibrous dysplasia with myxoma 53. Roediger 3rd HL, Karpicke JD. The power of testing memory: basic
(Mazabraud syndrome). Radiographics 2010; 30:827–832. doi:10.1148/ research and implications for educational practice. Perspect Psychol Sci
rg.303095155. 2006; 1:181–210. doi:10.1111/j.1745-6916.2006.00012.x.
29. Kransdorf MJ, Murphey MD. Diagnosis please. Case 12: Mazabraud syn- 54. Robins L. How to make your teaching more interactive. 2017. Available
drome. Radiology 1999; 212:129–132. doi:10.1148/radiology.212.1.r99jl26129. at: https://depts.washington.edu/dbpeds/Interactive_Teaching_Me-
30. Richardson ML, ed. Intergalactic online MSK conference: cool case book thods.docx. Accessed May 4, 2018.
1, Seattle, WA: BareBones Books; 2011. 55. Sterne GM, Richardson ML, Warren BH. Imaging findings in two cases of
31. Richardson ML, ed. Intergalactic online MSK conference: cool case book fluoroquinolone-induced Achilles tendinopathy. Radiol Case Rep 2006;
2, Seattle, WA: BareBones Books; 2011. 1:87–91. doi:10.2484/rcr.v1i3.32.
32. Richardson ML, ed. Intergalactic online MSK conference: cool case book 56. Akl EA, Pretorius RW, Sackett K, et al. The effect of educational games on
3, Seattle, WA: BareBones Books; 2013. medical students’ learning outcomes: a systematic review: BEME guide no
33. Richardson ML, ed. Intergalactic online MSK conference: cool case book 14. Med Teach 2010; 32:16–27. doi:10.3109/01421590903473969.
4: focus on nuclear medicine, Seattle, WA: Bare Bones Books; 2014. 57. Akl EA, Gunukula S, Mustafa R, et al. Support for and aspects of use of
34. Sharma N, Lau CS, Doherty I, et al. How we flipped the medical classroom. educational games in family medicine and internal medicine residency
Med Teach 2015; 37:327–330. doi:10.3109/0142159X.2014.923821. programs in the US: a survey. BMC Med Educ 2010; 10:26. doi:10.1186/
35. Network FL. Flip learning. 2017. Available at: http://flippedlearning.org. 1472-6920-10-26.
Accessed September 5, 2017. 58. Roubidoux MA, Chapman CM, Piontek ME. Development and evaluation
36. Hamdan N, McKnight P, McKnight K, et al. The flipped learning model: A of an interactive web-based breast imaging game for medical students.
white paper based on the literature review titled a review of flipped learn- Acad Radiol 2002; 9:1169–1178.
ing Flipped Learning Network/Pearson/George Mason University. 2013. 59. Dufenbach KR. Jeopardy for PowerPoint. 2017. Available at: https://
37. Alvarez B. Flipping the classroom: homework in class, lessons at home. sites.google.com/site/dufmedical/jeopardy. Accessed May 4, 2018.
Educ Digest 2012; 77:18. 60. Khan MN, Telmesani A, Alkhotani A, et al. Comparison of jeopardy game
38. Bergmann J, Sams A. Flip your classroom: reach every student in every format versus traditional lecture format as a teaching methodology in
class every day International Society for Technology in Education. 2012. medical education. Saudi Med J 2011; 32:1172–1176.
39. Wikipedia. Flipped classroom—Wikipedia, the free encyclopedia. 2017. 61. Shah BA, Sever A, Cressman S. Breast imaging bingo: an educational
Available at: https://en.wikipedia.org/w/index.php?title=Flipped_class- game to supplement the traditional assessment style. Med Sci Educ
room&oldid=788738274. Accessed September 5, 2017. 2016; 26:383–387.
40. Prober CG, Heath C. Lecture halls without lectures—a proposal for med- 62. Bochennek K, Wittekindt B, Zimmermann SY, et al. More than mere
ical education. N Engl J Med 2012; 366:1657–1659. doi:10.1056/ games: a review of card and board games for medical education. Med
NEJMp1202451. Teach 2007; 29:941–948. doi:10.1080/01421590701749813.
41. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped 63. Saxena A, Nesbitt R, Pahwa P, et al. Crossword puzzles: active learning
classroom” model of a renal pharmacotherapy module. Am J Pharm in undergraduate pathology and medical education. Arch Pathol Lab
Educ 2012; 76:196. doi:10.5688/ajpe7610196. Med 2009; 133:1457–1462. doi:10.1043/1543-2165-133.9.1457.

113