[go: up one dir, main page]

0% found this document useful (0 votes)
41 views11 pages

Rapid Prompting Method and Autism Spectrum Disorde

Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
41 views11 pages

Rapid Prompting Method and Autism Spectrum Disorde

Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/332956082

Rapid Prompting Method and Autism Spectrum Disorder: Systematic Review


Exposes Lack of Evidence

Article in Review Journal of Autism and Developmental Disorders · December 2019


DOI: 10.1007/s40489-019-00175-w

CITATIONS READS

22 5,275

10 authors, including:

Ralf Schlosser Howard c Shane


Northeastern University Boston Children's Hospital
153 PUBLICATIONS 5,660 CITATIONS 45 PUBLICATIONS 1,274 CITATIONS

SEE PROFILE SEE PROFILE

Russell Lang David Trembath


Texas State University Griffith University
232 PUBLICATIONS 9,808 CITATIONS 124 PUBLICATIONS 3,594 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Russell Lang on 24 June 2019.

The user has requested enhancement of the downloaded file.


Review Journal of Autism and Developmental Disorders
https://doi.org/10.1007/s40489-019-00175-w

REVIEW PAPER

Rapid Prompting Method and Autism Spectrum Disorder: Systematic


Review Exposes Lack of Evidence
Ralf W. Schlosser 1,2,3 & Bronwyn Hemsley 4 & Howard Shane 2,5 & James Todd 6 & Russell Lang 7 & Scott O. Lilienfeld 8,9 &
David Trembath 10 & Mark Mostert 11 & Seraphina Fong 12 & Samuel Odom 13

Received: 27 November 2018 / Accepted: 30 April 2019


# Springer Science+Business Media, LLC, part of Springer Nature 2019

Abstract
This systematic review is aimed at examining the effectiveness of the rapid prompting method (RPM) for enhancing motor,
speech, language, and communication and for decreasing problem behaviors in individuals with autism spectrum disorder
(ASD). A multi-faceted search strategy was carried out. A range of participant and study variables and risk and bias indicators
were identified for data extraction. RPM had to be evaluated as an intervention using a research design capable of empirical
demonstration of RPM’s effects. No studies met the inclusion criteria, resulting in an empty review that documents a meaningful
knowledge gap. Controlled trials of RPM are warranted. Given the striking similarities between RPM and Facilitated
Communication, research that examines the authorship of RPM-produced messages needs to be conducted.

Keywords Autism spectrum disorder . Facilitated communication . Rapid prompting method . Systematic review

The Centers for Disease Control and Prevention estimate that by deficits in social communication (American
1 in 59 children (1.7%) aged 8 years meet criteria for a diag- Psychological Association 2017) and often experience
nosis of autism spectrum disorder (ASD) and that this condi- marked difficulties with, or an absence of, spoken language
tion occurs in all ethnic, racial, and socio-economic groups (Mirenda and Iacono 2009). In fact, between 25 and 30% of
(Baio et al. 2018). Individuals with ASD are characterized children with ASD present with little or no functional speech

Electronic supplementary material The online version of this article


(https://doi.org/10.1007/s40489-019-00175-w) contains supplementary
material, which is available to authorized users.

* Ralf W. Schlosser 7
Special Education, Department of Curriculum and Instruction, Texas
R.Schlosser@northeastern.edu State University, 601 University Dr., San Marcos, TX 78666-4684,
USA
1 8
Department of Communication Sciences and Disorders, Northeastern Department of Psychology, Emory University, Room 473, Eagle
University, 360 Huntington Ave., Boston, MA 02115, USA Row, Atlanta, GA 30322, USA
2 9
Department of Otolaryngology and Communication Enhancement, School of Psychological Sciences, University of Melbourne,
Center for Communication Enhancement, Boston Children’s Melbourne, Victoria, Australia
Hospital, 9 Hope Ave., Waltham, MA, USA
10
3
Menzies Allied Health Institute, School of Allied Health Sciences,
Faculty of Humanities, Centre for Augmentative and Alternative Griffith University, 2.71_G40 Gold Coast Campus, Parklands Dr,
Communication, University of Pretoria, Pretoria, Gauteng, South Southport, Queensland 4215, Australia
Africa
11
4 School of Education, Regent University, 1000 University Drive,
Speech Pathology, Graduate School of Health, University of
Virginia Beach, VA 23454, USA
Technology Sydney, PO Box123, Broadway, Ultimo, NSW 2007,
Australia 12
Department of Psychology, Northeastern University, 360 Huntington
5
Otolaryngology, Harvard Medical School, Boston, MA, USA Ave., Boston, MA 02115, USA
6 13
Department of Psychology, Eastern Michigan University, Frank Porter Graham Child Development Institute, University of
Ypsilanti, MI, USA North Carolina, CB 8040, Chapel Hill, NC 27599-8040, USA
Rev J Autism Dev Disord

even after years of intervention (Anderson et al. 2009; Rose letters on a card. The technique presumes linguistic compe-
et al. 2016; Tager-Flusberg and Kasari 2013). These children tence (i.e., unconscious knowledge of grammar that allows a
are often introduced to augmentative and alternative commu- speaker to use and understand a language), and learning ma-
nication (AAC) interventions and systems to enable more ef- terials are presented verbally and textually at a chronological
fective communication (Mirenda and Iacono 2009). AAC in- age level rather than at a level of complexity informed by
terventions supplement or replace a child’s natural speech, assessment of the child’s developmental and language levels
handwriting, or both by using one or a combination of ges- (Mukhopadhyay 2008; Todd 2015).
tures, manual signs, communication boards and books, and
speech output technologies from dedicated speech-
A Brief History
generating devices or specialized applications on mobile tech-
nologies (Beukelman and Mirenda 2013; McNaughton and
The Cure Autism Now Foundation introduced RPM to the
Light 2013; Shane et al. 2012). Several systematic reviews
USA in 2001 (http://www.halo-soma.org). The foundation
and meta-analyses have concluded that, as a class, AAC inter-
website asserts that Soma Mukhopadhyay’s son (aged
ventions are effective in improving communication and
30 years), who was diagnosed with ASD at age three, was
communication-related skills in individuals with ASD (Allen
introduced to his mother’s Bown intensive educational
et al. 2017; Ganz 2015; Ganz et al. 2012a, b; Iacono et al.
curriculum … Activities included reading textbooks and
2016; Schlosser and Wendt, 2008). Subsets of AAC interven-
classics, prompting him to point to numbers and letters, and
tions or specific AAC modalities and strategies examined by
physically motoring his body through the motions.^ At six
systematic reviews include aided AAC approaches at large
years old, her son was said to write independently. In 2001,
(Ganz et al., 2012a, b), speech output technologies (Lorah
the Cure Autism Now Foundation offered Mukhopadhyay a
et al. 2015; Schlosser and Koul 2015), and the Picture
fellowship to implement her teaching method at a school in
Exchange Communication System (Flippin et al. 2010;
Los Angeles, working with nine children with ASD. Since
Ganz et al., 2012a, b).
then, Mukhopadhyay has refined and trademarked RPM and
trained Bhundreds of students throughout the United States^
(http://www.halo-soma.org).
Rapid Prompting Method

What Is It? Similarities and Differences between RPM and FC

Over the past few years, however, a superficially similar but RPM bears considerable similarities to Facilitated
markedly different technique, rapid prompting method Communication (FC) (aka Bsupported typing^), a technique
(RPM), has been gaining popularity. RPM is described by its whereby individuals with disabilities and communication im-
developer, Soma Mukhopadhyay, the Executive Director of pairments select letters by typing on a keyboard while receiv-
Education at Helping Autism for Learning and Outreach ing physical support (e.g., on the hand, arm, elbow, shoulder),
(HALO) as an instructional, therapeutic, and communication emotional encouragement,, and other communication sup-
technique (Mukhopadhyay 2008). Information about RPM is ports from facilitators (Institute for Community Inclusion,
available primarily through the HALO website and in n.d.). Both RPM and FC are facilitator-dependent techniques
Mukhopadhyay’s growing list of private and self-published designed for AAC alphabet/letter/word boards or speech-
RPM instructional books (Mukhopadhyay 2008, 2011, generating devices to support communication or education
2013, 2014, 2015, 2016a, b, 2017a, b). According to the (Tostanoski et al. 2013). In its standard form, RPM differs
HALO website, RPM is a Bbrain-based^ teaching and assis- from FC in that facilitators do not typically physically support
tive method that is aimed at establishing and promoting the child’s hand or arm. Rather facilitators hold and move a
pointing-, typing-, or writing-based textual communication letter board while the child makes selections, repositions the
in people with intellectual and developmental disabilities, board between selections, and provides ongoing repeated ver-
who are verbal or nonverbal and low functioning or high bal and gestural prompts for pointing to letters to spell out
functioning. It is further asserted that RPM is derived broadly words (Mukhopadhyay 2008; Todd 2015). However, the pos-
from the cognitive-developmental psychology of Jean Piaget sibility of facilitator influence on the message still exists.
and neuro-sensory theories of A. Jean Ayers and is aimed at Tostanoski et al. (2013) argued that Bit is irrelevant whether
remediating or accommodating problems in cognition and the facilitator’s unconscious bias and expectations are intro-
sensory sensitivities. Procedurally, RPM comprises a series duced via moving the person’s hand to touch the letter board,
of Bteach-ask^ trials of graduated difficulty, with the student as in FC, or moving the letter board to touch the person’s hand,
first being prompted to choose a correct answer from two as in RPM^ (p. 220). They go on to note that Mukhopadhyay
written options and later being prompted to point to printed (2008) does not address the fact that a letter board can be more
Rev J Autism Dev Disord

easily repositioned than the manipulation of someone else’s Rationale for the Present Systematic Review
hand or wrist.
Even though there is a standard form of RPM, it is not The impetus for conducting this review stems from several
unusual to see RPM facilitators touching individuals in ways considerations. First, to date, there is no systematic review
that could guide pointing. Todd’s (2015) analysis of training of RPM that would provide a higher level of scientific evi-
videos and depictions of RPM in popular movies revealed dence than individual studies. Second, ASD has been associ-
that, whether intended or unintended, facilitators may guide ated with a large number of popular but unsubstantiated inter-
responding by (a) offering a pointer on each trial close to the ventions, including FC (Lilienfeld et al. 2014). For example,
desired letter board target, (b) leaning against the client in a Smith’s (2008) review produced a list of 50 unsupported or
manner that could move the pointing hand, or (c) touching the weakly supported interventions for ASD, and a recent survey
individual’s thigh, back, or other body part in a way that could of 535 special educators of students with ASD and/or intellec-
provide pointing cues (Amora 2012; Mukhopadhyay 2008; tual disability revealed that RPM was used more than empir-
Pluktelevision 2010; Suefinnes 2013; Todd 2015). Ochs ically supported methods such as video modeling and the
et al. (2005), who conducted ethnographic research on RPM Picture Exchange Communication System (Knight et al.
use, also acknowledged the use of physical support. Finally, 2019). Likewise, recent surveys of early-intervention staff in
Mukhopadhyay (2008) acknowledged that Bif the teacher is in Australia (Paynter and Keen 2015; Paynter et al. 2017) sug-
a hurry to push through this stage, there is a risk of using gest that unsupported interventions, such as RPM, continue to
facilitated communication (when the teacher physically be used. Additionally, several parent surveys indicate that
touches the learner’s arm/elbow/shoulder while the learner is most children with ASD receive between four to seven inter-
communicating)^ (p. 192) in the process of transitioning to ventions (Schreck et al. 2014), many of which are unsubstan-
letter board usage. tiated and without evidence (Deyro et al., 2014; Nach 2009).
For FC, a corpus of 51 studies scrutinized across seven Furthermore, a study of variables influencing parental choices
reviews in peer-reviewed journals indicates that, when facili- of treatments revealed that parents were overwhelmed with
tator’s knowledge of the content to be communicated is con- confusing and often conflicting information regarding ASD
trolled, messages generated via FC are authored by the facil- treatment options (Frame 2014). Considering the widespread
itator (Felce 1994; Jacobson et al. 1995; Mostert 2001, 2010; use of RPM, the popularity of unsubstantiated interventions in
Probst 2005; Schlosser et al. 2014; Wehrenfennig and Surian the field of ASD at large, and the marked similarities between
2008). This finding strongly suggests that the alleged efficacy RPM and FC, a systematic review to inform the practice of
and effectiveness of FC is in fact attributable to the ideomotor pediatricians, child psychiatrists, child psychologists, special
effect (Hyman 1999), whereby facilitators inadvertently influ- educators, and others is warranted.
ence hand and finger movements of individuals with ASD as A third impetus relates to the increasing attention received
they facilitate (Jacobson et al., 1995). For RPM, the empirical by RPM in the popular media and via regulatory and political
question of who is authoring the messages remains discourse. For example, Senator Rónán Mullen advocated for
unresolved. the increased use of RPM to the Irish Senate in 2014,
portraying it as an effective method that has helped thousands
Previous Reviews of people over 10 years in the USA (Upper House of the Irish
Legislature, 2014). In February 2017, The Washington Post
Deacy et al. (2016) published the only review of RPM in the published an article on RPM juxtaposing the opinions of an
peer-reviewed literature. Based on the analysis of three studies increasing number of parents of children with ASD wishing to
(Chen et al. 2012; Gernsbacher 2004; Solomon 2006) they consider using RPM in schools with the views of scientists
concluded that (a) there appears to be an association between who expressed skepticism regarding its effectiveness as well
using RPM and a decrease in repetitive behavior and (b) as concern for user safety and autonomy (Chandler 2017).
Bfurther research in the area of RPM is warranted in order RPM has also been featured in several documentary films
for it to be considered an evidence-based practice^ (p. 92). including the 2010 film BA Mother’s Courage: Talking back
This review contained several methodological limitations to Autism.^
(Hemsley 2016), including (a) unclear review questions, (b) Finally, a systematic review is needed to support the role of
absence of design-related inclusion criteria (e.g., inclusion of professionals (e.g., pediatricians, speech-language patholo-
three descriptive studies incapable of validating efficacy), (c) gists, occupational therapists, and special educators) in
omission of a rigorous appraisal of study quality, and (d) lack assisting families of children with ASD to choose interven-
of identifying conflicts of interests related to funding sources. tions that are evidence based. Developmental and behavioral
Furthermore, their review did not delineate the specific disorders such as ASD are by far the most prevalent chronic
sources searched (e.g., databases, trial registers, and book conditions encountered by general pediatricians.
chapters). In sum, their review falls short of being systematic. Nevertheless, in the USA, there are only 750 board-certified
Rev J Autism Dev Disord

developmental-behavioral pediatricians (Bridgemohan et al. Information Clearinghouse (ProQuest), Google Scholar,


2018; https://www.abp.org/content/developmental- Linguistics and Language Behavior Abstracts (ProQuest),
behavioral-certification), who may be trained to discriminate Medline (Pubmed), ProQuest Dissertations and Theses
between validated evidence-based interventions and those Global, PsycINFO (ProQuest), Psychology and Behavioral
without evidence. In contrast, there are approximately Science Collection (ProQuest), and Web of Science. In addi-
4,000,000 infants born in the USA every year (https://www. tion, the following publisher-specific databases were
cdc.gov/nchs/fastats/births.htm), suggesting an incidence rate searched: Sage Journals Premier, ScienceDirect (Elsevier),
of 68,000 new births of children with ASD each year (i.e., SpringerLink Journals, and Taylor & Francis Online. To iden-
4,000,000 × 0.017). Thus, most children with ASD need to tify potentially unpublished studies or work in progress, the
be managed by their primary pediatric health care providers following trial registers were searched: EU Clinical Trial
(Patterson et al. 2012), and these and other providers need to Register (https://www.clinicaltrialsregister.eu), Australian
know which interventions for ASD are truly evidence based. New Zealand Clinical Trials Registry (http://www.anzctr.org.
Given that there is no cure for most developmental disorders, au/Default.aspx), ClinicalTrials.gov (https://clinicaltrials.gov)
understandably anxious, unsuspecting parents seeking ways , and ClinicalTrialSearch.org (https://www.clinicaltrialsearch.
to improve communicative competence are at significant risk org).
of pursuing often harmful non-evidence based treatments for Ancestry searches were conducted of studies considered
their children. In sum, parents, pediatricians, educators, for inclusion, as well as other relevant studies, reviews, book
speech-language pathologists, and other professionals who chapters, and books. Further, forward citation searches of in-
serve children with ASD are entitled to the best available cluded studies had been planned (via the Web of Sciences),
scientific evidence efficiently accessed via a systematic re- but not implemented because no studies qualified for inclu-
view that can establish the degree of empirical support for sion (see Results). Authors of screened articles were contacted
RPM. to request copies of unpublished or any other related literature.
We searched for the phrase BRapid Prompting Method^ as
Objective a free text phrase in title, abstract, and/or full text (as available
in bibliographic databases). Our preliminary search revealed
The aim of this systematic review was to determine whether that the search for BRPM^ as an acronym was overinclusive,
RPM is effective for individuals with ASD in terms of (a) yielding too many unrelated entries (e.g., revolutions per min-
improving body motor skills, speech, communication, ute). Further, our preliminary search revealed that an author
communication-related cognition, and/or (b) reducing prob- search for the developer of the RPM was feasible only if the
lem behavior (including ritualistic or repetitive behaviors). bibliographic database permitted the searching of the authors
Treatment comparisons were desirable but not necessary for full name (Mukhopadhyay) with at least the first initial (S.),
inclusion. otherwise there were too many unrelated entries. A prelimi-
nary search also showed that it was not necessary to combine
BRapid Prompting Method^ with any autism-related key-
Method words due to the relatively low number of relevant results.
Databases were also searched for Binformative pointing^ as
This systematic review was registered a priori with the a free text word in title, abstract, and/or full text (as available
PROSPERO database–International Prospective Register of in bibliographic databases) because it describes a similar if not
Systematic Reviews under the following registration number: identical method (Iversen 2007). We did not deem it necessary
PROSPERO 2016 CRD42016047871. The protocol can be to impose a temporal limit onto the search in terms of specific
accessed here: https://www.crd.york.ac.uk/PROSPERO/ years searched.
display_record.asp?ID=CRD42016047871.
Criteria for Inclusion and Exclusion
Search Methods
Inclusion criteria were as follows:
The original search was conducted in March 2017, repeated in
November 2017, and April 2018, and repeated once more on (a) Participants of any age with a diagnosis of ASD (i.e.,
September 30, 2018. A multi-faceted search strategy was autism prototype disorder, childhood disintegrative dis-
employed (Schlosser et al. 2005). The following databases order, Asperger disorder, pervasive developmental
were searched: Cochrane Library, ComDisDome (ProQuest), disabilities-not otherwise specified) regardless of co-
Cumulative Index of Nursing and Allied Health Literatures morbid diagnoses.
(EBSCO Host), Education Journals (ProQuest), Education (b) The study employed a (1) single-case experimental de-
Research Complete (EBSCO Host), Education Resources sign capable of establishing the effectiveness of RPM
Rev J Autism Dev Disord

(e.g., ABAB, alternating treatments design, adapted al- generalization assessed, (i) maintenance assessed, (i) inter-
ternating treatments design, multiple baseline design, observer agreement on outcome variables, (k) effect size
multiple probe design, parallel treatments design) or a (group designs), and (l) effect size indicators (single-case ex-
(2) group experimental design capable of establishing perimental designs).
the effectiveness of RPM (e.g., randomized controlled Data extraction from included studies was planned to occur
trials). blinded in the sense that the methods section of each study
(c) Group experimental designs that offered only group- was to be separated from the results and coded first, followed
level data and/or analyses required that all participants by the coding of the result-related sections. We intended to
have a diagnosis of ASD. calculate inter-rater agreement between the first and second
(d) Single-case experimental designs required that at least rater. Disagreements between the two reviewers were to be
one of the participants had a diagnosis of ASD and data resolved through consensus and, if necessary, the tie was to
were provided for that individual. be broken by a third reviewer.
(e) RPM was the intervention under investigation as evi-
denced by specific reference/s made to writings of its Risk of Bias (Quality) Assessment
developer Mukhopadhyay (2008) or any of the following
guides subsequently authored by Mukhopadhyay (2011, For group design studies, we intended to use the PEDro
2013, 2014, 2015, 2016a, b, 2017a, b). scale, which includes 11 items related to (a) subject eligi-
(f) The study was written in English (i.e., a preliminary bility, (b) random allocation, (c) allocation method, (d)
search in preparation for the PROSPERO registration baseline equivalence, (e) blinding of subjects, (f) blinding
revealed sources in English only), published in a of treatment agents, (g) blinding of assessors, (h) relation
peer-reviewed journal, and was available as an unpub- of # of allocated subjects to # of subjects from which out-
lished doctoral dissertation or master’s thesis, or a full comes were obtained, (i) between-group statistical compar-
manuscript length unpublished report (e.g., grant isons for at least one outcome, and (j) point and variability
report). measures for at least one outcome (http://www.pedro.org.
au/english/downloads/pedro-scale/). In addition, each
Exclusion criteria were as follows: group study was to be assessed for the availability of data
on treatment integrity.
(a) Studies involving participants described as Bautistic- For studies using single-case experimental designs, we
like^ or individuals whose Diagnosis was not confirmed. planned to use yes/no items from the EVIDAAC Single-
(b) Pre-experimental single-case designs (e.g., AB design, Subject Scale (Schlosser 2011) which is largely based on
ABA design) or pre-experimental group designs (e.g., Horner et al. (2005). Additionally, we planned to use the cer-
pretest-posttest design) because they do not permit the tainty of evidence framework originally developed by
establishment of a functional relation between interven- Simeonsson and Bailey (1991) and subsequently modified
tion and outcome. and applied in several systematic reviews (e.g., Granlund
(c) Studies in languages other than English. and Olsson 1999; Millar et al. 2006; Schlosser and Koul
(d) Abbreviated manuscripts such as brief abstracts pub- 2015; Schlosser and Sigafoos 2006; Schlosser and Wendt,
lished exclusively in conference proceedings, etc. 2008). This framework groups the quality of evidence accord-
ing to four categories by examining three features of a study:
design, inter-observer agreement of the dependent variable,
Study Selection and Data Extraction and treatment integrity: (a) conclusive, (b) preponderant, (c)
suggestive, and (d) inconclusive. For definitions of these cat-
The first and second authors independently applied the inclu- egories, please consult the PROSPERO protocol (https://
sion checklist to 100% of the retrieved records. Decisions www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=
(Bin^ or Bout^) were then compared and disagreements were CRD42016047871).
resolved through consensus building (see below).
The following data extraction procedures were planned but Strategy for Data Synthesis
not implemented because the review turned out to be empty.
Using a pilot-tested coding form, at least two review team For group designs, we intended to use a quantitative synthesis
members aimed to independently code all included studies (via Hedges’ g) provided that the data are sufficiently homog-
to extract the following data categories: (a) sample or n, (b) enous in terms of participant characteristics and outcomes
participant characteristics and demographics, (c) intervention targeted. This, however, was deemed unlikely to occur since
setting, (d) study design, (e) # of sessions per week, (f) fidelity we anticipated a relatively small number of included trials
of implementation, (g) types of outcomes targeted, (h) (n < 10) and relatively small number of participant in trials.
Rev J Autism Dev Disord

If quantitative synthesis was not going to be possible, we were located. After removing duplicates, 108 records were
aimed to report effect sizes per study and synthesize the results submitted for screening. After screening title and abstract,
narratively along with a tabular format. For single-case exper- 102 records were removed. Many of these records mentioned
imental designs, we aimed to use a quantitative synthesis pro- BRapid Prompting Method^ in the title, abstract, or body of
vided that the studies to be aggregated were sufficiently ho- the text without actually studying RPM as an intervention, for
mogenous in terms of participant characteristics and outcomes example, a discussion paper of (un)supported treatments in
targeted. If possible, we planned to use the following effect ASD. For the remaining 6 records, full texts were screened.
size indicators: (a) percentage of non-overlapping data None of the studies met the inclusion criteria.
(Schlosser et al. 2008; Scruggs et al. 1987); (b) improvement
rate difference (Parker et al. 2009); and (c) percent zero data
(only for interventions that is aimed at reducing problem Log of Rejected Studies/Papers and Primary Reasons
behavior; Scotti et al. 1991). If the data set had turned out for Rejection
too heterogeneous, we would have provided the above metrics
per study only in tabular format. Table 1 provides a summary of the reasons why the six
papers (Chen et al. 2012; Cleveland 2008; Gernsbacher
2004; Solomon 2006; Van Acker 2006; Van der Meer
Results et al. 2012) subjected to full-text review were excluded.
Several of the papers failed to meet design-related require-
Number of Studies Screened and Excluded ments (criterion # 2); descriptive designs such as case stud-
ies and ethnographic studies were not suitable for estab-
The PRISMA 2009 Flow Chart (see Fig. 1) (Moher et al. lishing effectiveness. Several manuscripts did not study
2009) illustrates the screening steps and associated number RPM as an intervention (criterion # 5) (see Table 1 for
of studies at each step of the process. Initially, 164 records exclusionary notes).

Fig. 1 PRISMA 2009 flow chart


PRISMA 2009
Flow Diagram
Identification

Records idenfied through Addional records idenfied


database searching through other sources
(n = 94) (n = 75)

Records aer duplicates removed


(n = 108)
Screening

Records screened Records excluded


(n = 108) (n = 102)

Full-text arcles assessed Full-text arcles excluded,


Eligibility

for eligibility with reasons


(n = 6) (n = 6)

Studies included in
qualitave synthesis
(n = 0)
Included

Studies included in
quantave synthesis
(meta-analysis)
(n = 0)
Rev J Autism Dev Disord

Table 1 Reasons for exclusion of papers that underwent full-text review

Paper–full-text Checklist item/s Exclusion notes


review not met

Chen et al. (2012) #2 Descriptive study


Cleveland (2008) #5 Although Mukhopadhyay’s work is reviewed in the literature section, the procedure described (p. 11) has
components of RPM but is not specifically RPM as described by Mukhopadyay
Gernsbacher # 2, # 5 # 2: Descriptive case study
(2004) # 5: The mother had met with Mukhopadhyay and as a result adapted some of her modalities. For example, she
tried handwriting as a form of AAC. It is stated she would not go through the procedures at great length as
described by the developer
Solomon (2006) # 2, # 5 # 2: Ethnographic study
# 5: Even though RPM was studied, it was examined as a practice rather than an intervention—comparing the
practices of RPM at HALO versus by parents elsewhere
Van Acker (2016) # 1, # 2, # 5 # 1, # 2: Discussion paper without participants or design
# 5: RPM was mentioned in the text but was not studied as an intervention
Van der Meer et al. # 5 RPM was mentioned in the text but was not studied as an intervention
(2012)

Inter-rater Agreement on Inclusion/Exclusion communication, communication-related cognition, and prob-


lem behaviors. Our review suggests that despite many years
To determine inter-rater agreement on inclusion and exclusion passing since RPM’s development, there is as yet no evidence
decisions, each record submitted to the inclusion checklist was for its efficacy or effectiveness.
independently reviewed by the first rater and the second rater. Empty reviews, which refer to an absence of evidence rath-
Proportional agreement on the primary inclusion decision er than evidence of absence of an effect, play a critical role in
(Bin^ or Bout^) was calculated by taking the number of agree- identifying gaps in knowledge and future research needs
ments divided by the number of agreements plus disagree- (Slyer 2016). Such reviews are premised on the legitimate
ments, multiplied by 100. We then calculated item-specific assumption that the onus of proof for novel interventions falls
inter-rater agreement, which involved responses to each of on their proponents (Pigliucci and Boudry 2014) given (a) the
the six items on the inclusion checklist per record. This pro- existence of empirically supported interventions for ASD and
cedure yielded an agreement of 91.18%. An analysis of initial (b) RPM’s similarities to a scientifically discredited technique,
disagreements revealed that many resulted from differential in this case, FC. Given that RPM involves facilitation closely
responding to whether an item should be rated as BN/A^ or akin to FC, the most critical need is to conduct studies to
BNo.^ A few disagreements were attributable to incorrect in- determine who is authoring the messages generated using
terpretation of the definitions. After consensus-building, these RPM. The descriptive studies on RPM to date (Chen et al.
disagreements were resolved, yielding 100% agreement. 2012; Gernsbacher 2004; Solomon 2006) along with their
review by Deacy et al. (2016) were predicated on the assump-
tion that the participants were the authors of the messages
Discussion generated, without first verifying that this was the case. In
light of the notable similarities of RPM with FC in terms of
After an exhaustive search and a rigorous process of applying facilitator behavior and the likelihood of ideomotor effects
a priori stated inclusion criteria, this systematic review yielded (Hyman 1999), this assumption is tenuous at best.
no evidence for or against the effectiveness of RPM for indi- Deacy et al. (2016) concluded their review by saying
viduals with ASD. A review in which no studies can be locat- that future research is needed before RPM can be identified
ed that meet pre-stated inclusion criteria is termed an Bempty as an evidence-based practice. Although we do not dis-
review^ (Higgins and Green 2011; Schlosser and Sigafoos agree with this conclusion, our empty review highlights
2009; Yaffe et al. 2012). Sometimes, empty reviews may arise that there is not a single study that demonstrates a causal
from very stringent design-related inclusion criteria such as relation between RPM and outcomes, and therefore RPM
allowing only randomized controlled trials (Yaffe et al. is utterly untested and not close to earning the status of an
2012) or from including only very narrow outcomes. evidence-based intervention. In a peculiar way, this state of
Nevertheless, this conclusion is not applicable here given that the literature is not entirely tragic because it offers an op-
any design that permitted documentation of effectiveness or portunity to first rule in or out whether RPM is
efficacy was considered. Relatedly, a wide range of dependent pseudoscientific—potentially saving valuable resources
variables were permitted including motor, speech, language, and minimizing potential harms.
Rev J Autism Dev Disord

If authorship studies were to determine that RPM is valid, it References


is imperative that future studies interrogating the effectiveness
of RPM in terms of social-communication and behavioral out- Allen, A. A., Schlosser, R. W., Brock, K. L., & Shane, H. C. (2017). The
effectiveness of aided augmented input technologies for persons
comes thoroughly assess and document the literacy skills of
with developmental disabilities: a systematic review. Augmentative
individuals prior to intervention. This desideratum is critical to and Alternative Communication, 33, 149–159.
ensure that any observed Beffects^ are not due simply to the American Psychological Association. (2017). Diagnostic and statistical
individual lacking appropriate opportunity, expectation, and manual of mental disorders. Washington, DC: CBS Publishing.
access to materials to demonstrate her or his existing literacy Amora, B. D. (2012). Nick with Soma [video file]. Retrieved from https://
www.youtube.com/watch?v=VRE0mRuj4dY. Accessed 10
skills, which could have been achieved through the provision Nov 2017.
of evidence-based AAC approaches. Additionally, future ef- Anderson, D. K., Oti, R. S., Lord, C., & Welch, K. (2009). Patterns of
fectiveness studies should (a) be aimed at meeting the inclu- growth in adaptive social abilities among children with autism spec-
sion criteria of this review, (b) meet accepted quality indica- trum disorders. Journal of Abnormal Child Psychology, 37(7),
1019–1034.
tors for the respective research design chosen, and (c) be in- Baio, J., Wiggins, L., Christensen, D. L., Maenner, M. J., Daniels, J.,
dependently replicated by more than one research team. Warren, Z., Kurzius-Spencer, M., Zahorodny, W., Robinson
Because there is no current objective scientific evidence for Rosenberg, C., White, T., Durkin, M. S., Imm, P. N., Yeargin-
RPM, the scientifically and ethically sound recommendation Allsopp, M., Lee, L. C., Harrington, R., Lopez, M., Fitzgerald, R.
T., Hewitt, A., Pettygrove, S., Constantino, J. N., Vehorn, A.,
is to not initiate its use until trials are completed and the inter- Shenouda, J., Hall-Lande, J., Van Naarden Braun, K., & Dowling,
vention is demonstrated to be safe (e.g., that it does not result N. F. (2018). Prevalence of autism spectrum disorder among chil-
in uncorroborated accusations of sexual abuse against family dren aged 8 years — autism and developmental disabilities moni-
members, a known risk of FC) and effective. toring network, 11 Sites, United States, 2014. CDC MMWR
Surveillance Summaries, 67(SS-6), 1–23.
Beukelman, D. R., & Mirenda, P. (2013). Augmentative and alternative
Limitations communication: supporting children and adults with complex com-
munication needs. Baltimore, MD: Paul H. Brookes Publishing Co.
Bridgemohan, C., Bauer, N. S., Nielsen, B. A., DeBattissta, A., Ruch-
We included studies written in English only and our search Ross, H. S., Paul, L. B., & Roizen, N. (2018). A workforce survey
efforts did not explicitly target non-English sources. Although on developmental-behavioral pediatrics. Pediatrics, 141(3),
RPM was developed in the English-speaking context (i.e., e20172164.
Chandler, M. A. (2017). Parents want to give their autistic children a
India and subsequently the USA (Todd 2015)) and our pre-
voice in the schools, but scientists call their technique ‘false hope’.
liminary search had revealed no non-English sources, we can- The Washington Post Retrieved from https://www.washingtonpost.
not rule out a language bias. com/local/social-issues/parents-of-autistic-children-are-pushing-
schools-to-allow-controversial-communication-techniques/2017/
02/28/1bd33da2-ed6a-11e6-9973-c5efb7ccfb0d_story.html?
noredirect=on&utm_term=.c658118d94db. Accessed 31 Dec 2017.
Conclusions Chen, G. M., Yoder, K. J., Ganzel, B. L., Goodwin, M. S., & Belmonte,
M. K. (2012). Harnessing repetitive behaviors to engage attention
and learning in a novel therapy for autism: an exploratory analysis.
This systematic review was deemed empty and revealed no Frontiers in Psychology, 16(3), 1–16.
evidence in relation to RPM’s effectiveness meeting our in- Cleveland, J. (2008). The effects of multiple prompting on acquisition
clusion criteria. The developers of RPM and its proponents training for individuals with intellectual disabilities (Master’s the-
sis) (order no. 1456403). Available from ProQuest Dissertations &
have yet to fulfill the crucial burden of proof requirement
Theses Global. (219949961). Retrieved from http://ezproxy.neu.
demanded for novel interventions. Although lack of evidence edu/login?url=https://search-proquest-com.ezproxy.neu.edu/
does not necessarily demonstrate a lack of effect, until future docview/219949961?accountid=12826.
trials have demonstrated safety and effectiveness, and perhaps Deacy, E., Jennings, F., & O’Halloran, A. (2016). Rapid prompting meth-
more importantly, have first clarified the authorship question, od (RPM): a suitable method for students with ASD? REACH
Journal of Special Needs Education Ireland, 29(2), 92–100.
we strongly discourage clinicians, educators, and parents of Deyro, M. C., Simon, E. W., & Guay, J. (2014). Parental awareness of
children with ASD from using RPM. empirically established treatments for autism spectrum disorders.
Focus on Autism and Other Developmental Disorders, 31(3), 184–
Compliance with Ethical Standards 195.
Felce, D. (1994). Facilitated communication: results from a number of
recently published evaluations. British Journal of Learning
Conflict of Interest The authors declare that they have no conflict of Disabilities, 22(4), 122–126.
interest.
Flippin, M., Reszka, S., & Watson, L. R. (2010). Effectiveness of the
picture exchange communication system (PECS) on communication
Ethical Approval This article does not contain any studies with human and speech for children with autism spectrum disorders. American
participants or animals performed by any of the authors. Journal of Speech-Language Pathology, 19(2), 178–195.
Frame, K. N. (2014). An analysis of variables influencing parental
Informed Consent N/A. choices of treatments for their child with autism spectrum disorders
Rev J Autism Dev Disord

(Unpublished doctoral dissertation). University of Memphis, van der Meer, L., Kagohara, D., Achmadi, D., O’Reilly, M. F., Lancioni,
Memphis, TN. G. E., Sutherland, D., & Sigafoos, J. (2012). Speech-generating
Ganz, J. B. (2015). AAC interventions for individuals with autism spec- devices versus manual signing for children with developmental dis-
trum disorders: state of the science and future research directions. abilities. Research in Developmental Disabilities, 33(5), 1658–
Augmentative and Alternative Communication, 31(3), 203–214. 1669.
Ganz, J. B., Davis, J. L., Lund, E. M., Goodwyn, F. D., & Simpson, R. L. Millar, D., Light, J. C., & Schlosser, R. W. (2006). The impact of aug-
(2012a). Meta-analysis of PECS with individuals with ASD: inves- mentative and alternative communication intervention on the speech
tigation of targeted versus non-targeted outcomes, participant char- production of individuals with developmental disabilities: a research
acteristics, and implementation phase. Research in Developmental review. Journal of Speech, Language, and Hearing Research, 49,
Disabilities, 33(2), 406–418. 248–264.
Ganz, J. B., Earles-Vollrath, T. L., Heath, A. K., Parker, R. I., Rispoli, M. Mirenda, P., & Iacono, T. (2009). Autism spectrum disorders and AAC.
J., & Duran, J. B. (2012b). A meta-analysis of single case research Baltimore: Paul H. Brookes Publishing Co.
studies on aided augmentative and alternative communication sys- Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA
tems with individuals with autism spectrum disorders. Journal of Group. (2009). Preferred reporting items for systematic reviews
Autism and Developmental Disorders, 42(1), 60–74. and meta-analyses: the PRISMA statement. PLoS Medicine, 6(7),
Gernsbacher, M. A. (2004). Language is more than speech: a case study. e1000097. https://doi.org/10.1371/journal.pmed.1000097.
Journal of Developmental Learning Disorders, 8, 81–98. Mostert, M. P. (2001). Facilitated communication since 1995: a review of
Granlund, M., & Olsson, C. (1999). Efficacy of communication interven- published studies. Journal of Autism and Developmental Disorders,
tion for presymbolic communicators. Augmentative and Alternative 31(3), 287–313.
Communication, 15, 25–37. Mostert, M. (2010). Facilitated communication and its legitimacy —
Hemsley, B. (2016). Evidence does not support the use of rapid twenty-first century developments. Exceptionality, 18(1), 31–41.
prompting method (RPM) as an intervention for students with au- Mukhopadhyay, S. (2008). Understanding autism through rapid
tism spectrum disorder and further primary research is not justified. prompting method. Denver: Outskirts Press.
Evidence-Based Communication Assessment and Intervention, Mukhopadhyay, S. (2011). Curriculum guide for autism using rapid
10(3–4), 122–130. prompting method: with lesson plan suggestions. Denver:
Outskirts Press.
Higgins, J. P. T., & Green, S. (Editors) (2011). Cochrane handbook for
Mukhopadhyay, S. (2013). Developing communication for autism using
systematic reviews of interventions Version 5.1.0. The Cochrane
rapid prompting method. Denver: Outskirts Press.
Collaboration. Available from http://handbook.cochrane.org.
Mukhopadhyay, S. (2014). Developing motor skills for autism using rap-
Accessed March 2011.
id prompting method: steps to improving motor function. Denver:
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M.
Outskirts Press.
(2005). The use of single-subject research to identify evidence-
Mukhopadhyay, S. (2015). Harnessing stims and behaviors in autism
based practices in special education. Exceptional Children, 71(2),
using rapid prompting method. Denver: Outskirts Press.
165–179.
Mukhopadhyay, S. (2016a). Developing expressive language in verbal
Hyman, R. (1999). The mischief-making of ideomotor action. Scientific
students with autism using rapid prompting method. Denver:
Review of Alternative Medicine, 3(2), 34–43.
Outskirts Press.
Iacono, T., Trembath, D., & Ericsson, S. (2016). The role of augmentative Mukhopadhyay, S. (2016b). Mathematical concepts for autism using
and alternative communication for children with autism: current rapid prompting method. Denver: Outskirts Press.
status and future trends. Neuropsychiatric Disease and Treatment, Mukhopadhyay, S. (2017a). Developing the visual skill of reading using
12, 2349–2361. rapid prompting method. Amazon Digital Services LLC.
Iversen, P. (2007). The informative pointing method (video file). Mukhopadhyay, S. (2017b). Learning life skills using rapid prompting
R e t r i e v e d f r o m h t t p s : / / w w w. y o u t u b e . c o m / w a t c h ? v = method-part 1. Amazon Digital Services LLC.
7m351faqO7Y. Accessed 10 Nov 2017. Nach, E. J. (2009). Instructional use of research-based practices for stu-
Jacobson, J. W., Mulick, J. A., & Schwartz, A. A. (1995). A history of dents with autism spectrum disorders (unpublished doctoral disser-
facilitated communication: Science, pseudoscience, and antiscience tation). Lynn University, Boca Raton.
science working group on facilitated communication. The American Ochs, E., Solomon, O., & Sterponi, L. (2005). Limitations and transfor-
Psychologist, 50(9), 750–765. mations of habitus in child-directed communication. Discourse
Knight, V. F., Huber, H. B., Kuntz, E. M., Carter, E. W., & Juarez, A. P. Studies, 7(4–5), 547–583.
(2019). Instructional practices, priorities, and preparedness for edu- Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate
cating students with autism and intellectual disability. Focus on difference for single-case research. Exceptional Children, 75(2),
Autism and other Developmental Disorders, 34(1), 3–14. https:// 135–150.
doi.org/10.1177/1088357618755694. Patterson, S., Smith, V., & Jelen, M. (2012). Autism spectrum disorders:
Lilienfeld, S. O., Marshall, J., Todd, J. T., & Shane, H. C. (2014). The information for pediatricians supporting families of young children
persistence of fad interventions in the face of negative scientific on the spectrum. In S. I. Ahmad (Ed.), Neurodegenerative diseases.
evidence: facilitated communication for autism as a case example. Advances in experimental medicine and biology (pp. 51–60). New
Evidence-Based Communication Assessment and Intervention, 8(2), York: Springer.
62–101. Paynter, J. M., & Keen, D. (2015). Knowledge and use of intervention
Lorah, E. R., Parnell, A., Schaefer, A., Whitby, P., & Hantula, D. J. practices by community-based early intervention service providers.
(2015). A systematic review of tablet computers and portable media Journal of Autism and Developmental Disorders, 45(6), 1614–
players as speech generating devices for individuals with autism 1623.
spectrum disorders. Journal of Autism and Developmental Paynter, J. M., Ferguson, S., Fordyce, K., Joosten, A., Paku, S., Stephens,
Disorders, 45, 3792–3804. M., Trembath, D., & Keen, D. (2017). Utilisation of evidence-based
McNaughton, D., & Light, J. C. (2013). The iPad and mobile technology practices by ASD early intervention service providers. Autism: The
revolution: benefits and challenges for individuals who require aug- International Journal of Research and Practice, 21(2), 167–180.
mentative and alternative communication. Augmentative and Pigliucci, M., & Boudry, M. (2014). Prove it! The burden of proof game
Alternative Communication, 29, 106–116. in science vs. pseudoscience disputes. Philosophia, 42(2), 487–502.
Rev J Autism Dev Disord

Pluktelevision (2010). Autism workshop: informative pointing – section Shane, H. C., Laubscher, E., Schlosser, R. W., Flynn, S., Sorce, J., &
1: part 1 of 4 [video file]. Retrieved from https://www.youtube.com/ Abramson, J. (2012). Applying technology to visually support lan-
watch?v=03F-Gn7oc6M. Accessed 10 Nov 2017. guage and communication in individuals with ASD. Journal of
Probst, P. (2005). BCommunication unbound – or unfound^? Ein integra- Autism and Developomental Disorders, 42, 1228–1235.
tives Literatur-Review zur Wirksamkeit der ‘Gestützten Simeonsson, R., & Bailey, D. (1991). Evaluating programme impact:
Kommunikation’ (‘Facilitated Communication/FC’) bei levels of certainty. In D. Mitchell & R. Brown (Eds.), Early inter-
nichtsprechenden autistischen und intelligenzgeminderten vention studies for young children with special needs (pp. 280–296).
Personen. Zeitschrift für Klinische Psychologie, Psychiatrie und London: Chapman and Hall.
Psychotherapie, 53(2), 93–128. Slyer, J. T. (2016). Unanswered questions: implications of an empty re-
Rose, V., Trembath, D., Keen, D., & Paynter, J. (2016). The proportion of view. JBI Database of Systematic Reviews and Implementation
minimally verbal children with autism spectrum disorder in a Reports, 14(6), 1–2.
community-based early intervention programme. Journal of Smith, T. (2008). Empirically supported and unsupported treatments for
Intellectual Disabilities Research, 60(5), 464–477. autism spectrum disorders. Scientific Review of Mental Health
Schlosser, R. W. (2011). EVIDAAC single-subject scale. Unpublished Practice, 6, 3–20.
manuscript, Northeastern University, Boston. Solomon, O. (2006). The Brapid prompting^ Method of communicating
Schlosser, R. W., & Koul, R. (2015). Speech output technologies in in- with severely autistic children: A language socialization study. Final
terventions for individuals with autism spectrum disorders: a scop- report. Washington, DC: U.S. National Academy of Education.
ing review. Augmentative and Alternative Communication, 31, 285–
Suefinnes (2013). RPM all that people deserve is ..talking about his
309.
autism [video file]. Retrieved from https://www.youtube.com/
Schlosser, R. W., & Sigafoos, J. (2006). Augmentative and alternative
watch?v=VUpfh9KrSQo
communication interventions for persons with developmental dis-
abilities: narrative review of comparative single-subject experimen- Tager-Flusberg, H., & Kasari, C. (2013). Minimally verbal school-aged
tal studies. Research in Developmental Disabilities, 27, 1–29. children with autism: the neglected end of the spectrum. Autism
Schlosser, R. W., & Sigafoos, J. (2009). Empty’ reviews and evidence- Research, 6, 468–478.
based practice. Evidence-Based Communication Assessment and Todd, J. T. (2015). Old horses in new stables: rapid prompting, facilitated
Intervention, 3, 1–3. communication, science, ethics, and the history of magic. In R. M.
Schlosser, R. W., & Wendt, O. (2008). Effects of augmentative and alter- Foxx & J. A. Mulick (Eds.), Controversial therapies for develop-
native communication intervention on speech production in children mental disabilities: fad, fashion, and science in professional
with autism: a systematic review. American Journal of Speech- practice (pp. 372–409). Mahwah: Rutledge.
Language Pathology, 17, 212–230. Tostanoski, A., Lang, R., Raulston, T., Carnett, A., & Davis, T. (2013).
Schlosser, R. W., Wendt, O., Angermeier, K., & Shetty, M. (2005). Voices from the past: comparing the rapid prompting method and
Searching for and finding evidence in augmentative and alternative facilitated communication. Developmental Neurorehabilitation,
communication: navigating a scattered literature. Augmentative and 17(4), 219–223.
Alternative Communication, 21, 233–255. Upper House of the Irish Legislature. (2014). Speech on autism support
Schlosser, R. W., Lee, D. L., & Wendt, O. (2008). Application of the services by Rónón Mullen. Parliamentary Debates, Official Report,
percentage of non-overlapping data in systematic reviews and me- 235, 9.
ta-analyses: a systematic review of reporting characteristics. Van Acker, R. (2006). Outlook on special education practice. Focus on
Evidence-Based Communication Assessment and Intervention, 2, Exceptional Children, 38(8), 8–18.
163–187. Wehrenfennig, A., & Surian, L. (2008). Autismo e comunicazione
Schlosser, R. W., Balandin, S., Hemsley, B., Iacono, T., Probst, P., & von facilitata: Una rassegna degli studi sperimentali. Psicologia
Tetzchner, S. (2014). Facilitated communication and authorship: a Clinica dello Sviluppo, 12, 437–464.
systematic review. Augmentative and Alternative Communication, Yaffe, J., Montgomery, P., Hopewell, S., & Shepard, L. D. (2012). Empty
30(4), 359–368. reviews: a description and consideration of Cochrane systematic
Schreck, K. A., Guide, C., & York, N. Y. (2014). Autism, parents, and r e v i e w s wi t h n o i n c l u d e d s t ud i e s . P L o S ON E , 7 ( 5 ) ,
treatments for their children. In V. Patel, V. Preedy, & C. Martin e36626. https://doi.org/10.1371/journal.pone.0036626.
(Eds.), Comprehensive guide to autism (pp. 2283–2296). New
York: Springer.
Publisher’s Note Springer Nature remains neutral with regard to
Scotti, J. R., Evans, I. M., Meyer, L., & Walker, P. (1991). A meta-
jurisdictional claims in published maps and institutional affiliations.
analysis of intervention research with problem behavior: treatment
validity and standards of practice. American Journal of Mental
Retardation, 96(3), 233–256.
Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative
analysis of single subject research methodology: methodology and
validation. Remedial and Special Education, 8, 24–33.

View publication stats

You might also like