Privatization The-World-Bank-Research-Observer-33-1 PDF

Public Disclosure Authorized
February 2018
INSIDE
T H E W O R L D B A N K
The Whys of Social Exclusion:
Insights from Behavioral
1818 H Street NW Economics
Washington, DC 20433, USA
Downloaded from https://academic.oup.com/wbro/issue/33/1 by Joint Bank-Fund Library user on 09 August 2019

World Wide Web: http://www.worldbank.org/ Generalization in the Tropics –
E-mail: researchobserver@worldbank.org Development Policy,

The World Bank Research Observer
Randomized Controlled Trials,
and External Validity
Privatization in Developing
Countries: What Are the
T H E W O R L D B A N K Lessons of Recent Experience?
Research
Public-Private Partnerships
in Developing Countries: The
Emerging Evidence-based
Critique
Observer
Volume 33 • Number 1 • February 2018
Volume 33, Issue 1
ISBN 978-0-19-880922-7
ISSN 0257-3032 (PRINT)

9 780198 809227 ISSN 1564-6971 (ONLINE)
academic.oup.com/wbro
WBRO_33_1_Cover.indd 1 20/03/18 9:13 PM

THE WORLD BANK
Research Observer
Volume 33 r Number 1 r February 2018
The Whys of Social Exclusion: Insights from Behavioral

Economics
Karla Hoff and James Walsh 1
Generalization in the Tropics – Development Policy, Randomized

Controlled Trials, and External Validity
Jörg Peters, Jörg Langbein, and Gareth Roberts 34
Privatization in Developing Countries: What Are the Lessons of

Recent Experience?
Saul Estrin and Adeline Pelletier 65
Public-Private Partnerships in Developing Countries: The

Emerging Evidence-based Critique
James Leigland 103
The Whys of Social Exclusion: Insights
from Behavioral Economics
Downloaded from https://academic.oup.com/wbro/article-abstract/33/1/1/4951690 by World Bank and IMF user on 08 August 2019
Karla Hoff and James Walsh
All over the world, people are prevented from participating fully in society through mecha-
nisms that go beyond the structural and institutional barriers that rational choice theory
identifies (—poverty, exclusion by law or force, taste-based or statistical discrimination,
and externalities from social networks differentiated by socioeconomic status). This paper
discusses four additional mechanisms that can be explained by bounded rationality: (a) im-
plicit discrimination, (b) self-stereotyping and self-censorship, (c) rules of thumb adapted
to disadvantaged environments that are dysfunctional in more privileged settings, and (d)
“adaptive preferences,” in which an excluded group comes to view its exclusion as natural.
Institutions, if they are stable, come to have cognitive foundations—concepts, categories,
social identities, and worldviews—through which people mediate their perceptions of them-
selves and the world around them. Abolishing or reforming a discriminatory institution may
have little effect on the social categories it created; groups previously discriminated against
by law may remain excluded through custom and habits of the mind. Recognizing new forces
of social exclusion, behavioral economics identifies ways to offset them. Some interventions
have had very consequential impacts.
All over the world, people are prevented from participating fully in society for reasons
that go beyond the structural and institutional barriers that rational choice theory
identifies—poverty, exclusion by law or force, taste-based or statistical discrimina-
tion, and externalities from social networks differentiated by socioeconomic status.1
As the precision of economics has increased through field, lab, and as-if-random
“natural” experiments, researchers have uncovered socio-psychological barriers to
upward mobility. In India, low-caste boys solved mazes just as well as high-caste boys
when their caste was not publicly revealed, but solved 23 percent fewer mazes than
high-caste boys when caste identity was revealed in mixed-caste groups (Hoff and
Pandey 2014). In France, grocery store clerks of African origin were 9 percent more
© The Author(s) 2018. Published by Oxford University Press on behalf of the International Bank for Reconstruction and
Development / THE WORLD BANK. All rights reserved. For permissions, please e-mail journals.permissions@oup.com
doi: 10.1093/wbro/lkx010 33:1–33
productive than other clerks except on days when they were supervised by managers
implicitly biased against minorities; on these days, they were of average productivity
(Glover, Pallais, and Pariente 2017). A belief that a race, gender, caste, or other
ascriptive group is inferior can affect how others treat members of the group and
how members of the group feel about themselves, creating productivity differences
that sustain the beliefs, although no inherent productivity differences exist (Hoff and
Stiglitz 2010; World Bank 2015).
In the last 30 years, economics has taken a cognitive turn. Economists and psychol-
ogists have made breakthroughs in understanding how people make decisions, and a
new field has emerged—behavioral economics. Camerer (2005) and Hoff and Stiglitz
(2016) distinguish two strands. With insights from psychology, Strand 1 views the in-
dividual as a quasi-rational actor: he thinks clearly under ideal conditions, but under
most real-world conditions his judgment and behavior are affected by seemingly irrel-
evant contextual factors in the moment of decision (Thaler 2016). With insights from
social psychology, sociology, and anthropology, Strand 2 views the individual as a
quasi-rational, enculturated actor: experience and exposure to social patterns have per-
sistent effects on judgment and behavior by shaping the cognitive toolkit with which
information is processed. The toolkit includes categories, concepts, causal narra-
tives, and other mental models (or, equivalently, schemas; Douglas 1966, 1986; Bruner
1991; D’Andrade 1995; Strauss and Quinn 1997; Bicchieri 2006). An individual
may have multiple, inconsistent mental models to interpret a situation; and cues
in the environment will influence which one is activated (DiMaggio 1997, p. 275).
There is a two-way relationship between individuals and institutions. Individuals cre-
ate institutions, but institutions shape the mental models of individuals and what
primes particular behaviors: “In an ongoing cycle of mutual constitution, people are
socioculturally shaped shapers of their environment” (Markus and Kitayama 2010).
The purpose of this essay is to provide a perspective from behavioral economics on
the forces that maintain social exclusion and on interventions to offset them. The es-
say is divided into five sections. We first provide evidence that institutions influence
how people think. Then we discuss four mechanisms through which an institution
that excludes an ascriptive group can have persistent effects long after the institution
is reformed or abolished. The mechanisms for this exclusion are (a) implicit discrimi-
nation, (b) self-stereotyping and self-censorship, (c) the rules of thumb of individuals
who try to live in two worlds—a narrow and insecure world of disadvantaged groups
and an orderly world of school or work, and (d) “adaptive preferences,” in which the
excluded group comes to view the exclusion as natural. We discuss interventions to
offset each mechanism, in some cases with hugely consequential impact.
The “Schematizing Power of Institutions”2

A common definition of institutions is “rules of the game.” In traditional economics,
the rules (in particular, property rights) affect only the opportunity sets over which
2 The World Bank Research Observer, vol. 33, no. 1 (February 2018)
people optimize; institutions have no impact on the way individuals think. In contrast,
all other social sciences recognize that institutions have a “schematizing power”: they
shape the knowledge structures (mental models or schemas) that an individual uses
to process information—what he attends to, what he perceives, and how he interprets
ambiguous signals. Bruner (1990) argues that the schematizing power of institutions
creates and sustains social identities: the symbolic systems make the user a reflection
of the community by influencing how others see him and how he sees himself.
Social groups that differ in their experiences and exposure have different men-
tal models and behave in systematically different ways in the same situation (e.g.,
Henrich et al. 2001; Brooks, Hoff, and Pandey, 2017). Many scholars view shared
mental models as a primary manifestation of culture (see Douglas 1986, espe-
cially pp. 46–48; Swidler 1986; and the definition in DiMaggio 1997, of “cul-
ture as a network of interrelated schemata”). Culture has a constitutive role,
not merely a regulatory role. Economists and political scientists increasingly in-
corporate mental models or rules of thumb (based on a schematic view of a
situation) as a variable to explain change or persistence of inequality; for ex-
ample, Hoff, Fehr, and Kshetramade 2011 (impact of caste identity on altru-
istic punishment in India to protect an ingroup); Alesina, Giuliano, and Nunn
2013 (impact of plough cultivation in pre-industrial agriculture on modern gen-
der roles); Acharya, Blackwell, and Sen 2016 (impact of the level of historical de-
pendence on slave labor on contemporary racism in the United States); Bedolla and
Miachelson 2012 and Carpenter and Foos 2017 (impact of “learned disengagement”
of marginalized U.S. citizens on the response to get-out-the-vote activities).
Categories, one kind of mental model, lay the foundation for social stratification.
The psychologist Gordon Allport (1950) argued that “[t]he human mind must think
with the aid of categories. Once formed, categories are the basis for normal prejudg-
ment. We cannot possibly avoid this process. Orderly living depends on it.” Institu-
tions that create hierarchies of ascriptive groups (e.g., by race, gender, or ethnicity)
impair the ability of others to learn things about the person that do not fit the cate-
gory, since mental models filter information in a way that tends to preserve categorical
beliefs. Psychologists are beginning to understand the neural basis of categorization
and associative learning:
When neurons are consistently activated by co-occurring features of experience,
physical changes in the neurons strengthen the connections between and among
them…Thereafter, if one of those neurons is activated, it will be more likely to ac-
tivate another in that group … Growing up in an environment of a given cultured
shape brings with it a distinctive pattern of experiences and corresponding neural
changes…. The synaptic changes… cannot be erased like sentences from a text… Change
in the world can lead to a new pattern of strong neural connections, but it does
not completely destroy earlier learning (Strauss and Quinn 1997, 90; emphasis
added).
Hoff and Walsh 3

A simple experiment by Bruner and Potter (1964) in visual identification provides
suggestive evidence that categorization leads individuals to resist disconfirming evi-
dence. Subjects were shown eight photographs of ordinary objects (e.g., a dog stand-
ing on grass), projected one at a time, which were gradually brought into focus. One
group of subjects saw the photos starting from almost complete blur. Another group
of subjects saw the photos starting from medium blur. And a third group saw the
photos starting from light blur. For all groups, the picture being exposed was stopped
at the same point of focus, regardless of its starting point. At this common termi-
nal point, the subject was asked what the object was. The surprising finding was that
subjects who had seen the longer video, starting at greater blur, were less likely to
identify the object correctly; that is, despite more exposure, they learned less. Slightly
less than one-fourth of the subjects recognized pictures when they began their view-
ing with a very blurred image, but more than half recognized them when viewing
began with light blur. Bruner and Potter (1964) suggest that people have difficulty
rejecting mental representations that they have constructed. Individuals “hang on”
to false hypotheses: “at any particular clarity of the display, those who see it for the
first time are more likely to recognize the object than those who started viewing at a
less clear stage.” This is called an interference phenomenon.
Kahan et al. (2017) demonstrate a related finding in the political domain. The au-
thors find that when people process scientific data that conflicts with their ingrained
worldviews (e.g., that gun control increases crime), they often misinterpret the data.
The interference problem is more severe, the more numerate they are. This indi-
cates that the problem is not inadequate mathematical skill. The authors suggest that
mathematical skill may actually enhance the ability to filter out unwanted informa-
tion.
People may be capable of suppressing their biases in clear-cut cases but incapable
of doing so in situations of ambiguity. The psychologists John Darley and Paget Gross
(1983) investigated how the social class of a student influences others’ judgments of
how well she is doing in school. The experimental subjects were randomly allocated
to one of four groups. Group 1 saw a video of a nine-year old girl, called Hannah,
in a low-income neighborhood and were informed that her parents had only a high
school education. Group 2 saw a video of Hannah in a high-income neighborhood
and were informed that her parents were college-educated. Groups 3 and 4 had the
same information as groups 1 and 2, respectively, but in addition viewed a videotape
depicting Hannah taking an oral test. There was only one version of the videotape.
It depicted Hannah’s performance as inconsistent—she answered some challenging
questions correctly and some easy questions incorrectly.
Traditional economics would predict that additional information could only in-
crease the precision of participants’ assessments and narrow the gap in assessments
between them, but the opposite was true. The first two groups, who had informa-
tion only about Hannah’s socioeconomic background, differed very little in their
assessment of how well she was doing in school. In contrast, groups 3 and 4
differed significantly in their assessments of how well Hannah was doing in school.
The “rich” Hannah was judged to be more able than the “poor” Hannah. Expecting
her to do better, viewers of the “rich” Hannah compared to viewers of the “poor”
Hannah evaluated her performance in the oral test more favorably. The results sup-
ported the hypothesis that mental models play a role in information-processing that
is distinct from their role in pre-judgment.
Alesina, Guiliano, and Nunn (2013) provide an example of the persistent effect,
throughout the world, of the meanings that an historical institution gave to gender.
Pre-industrial agriculture used either shifting cultivation or plough cultivation. Un-
like the hoe or digging stick used to prepare the soil in shifting cultivation, the plough
requires significant strength—either to pull it or to control the animal that pulls it.
In areas topographically well-suited to crops for which plough agriculture is efficient
(wheat, barley, and rye), men had an advantage in farming relative to women, and
adoption of the plough created gendered occupations—men in the field, women at
home—that have influence in modern times. In such areas, female labor force partic-
ipation in the year 2000 was more than 20 percentage points lower than in other ar-
eas. This influence remains as individuals migrate: the gender norms of immigrants’
children who live in the United States and Europe are influenced by whether their
ancestors were members of an ethnic group that used plough cultivation in the pre-
industrial period.
Mental models that an institution of social exclusion creates can be deliberately
strengthened after the institution is abolished in order to make the old social pattern
persist. In U.S. southern counties that in 1860 had a high proportion of slaves, whites
are more likely today to express racial resentment toward African Americans and to
oppose affirmative action, compared to whites who live in otherwise similar areas that
had lower population shares of slaves (Acharya, Blackwell, and Sen 2016). In order to
hold down agricultural labor costs after slavery was abolished (as well as for social and
political reasons), whites in counties that had relied heavily on slave labor reinforced
racist norms and racial hostility. This shaped attitudes that were transmitted across
generations through culture and institutions, such as Jim Crow. Anti-black attitudes
faded earlier in areas with a low historical dependence on slave labor.
Drawing on many other social sciences, twenty-first century behavioral economics
(which we have called Strand 2) introduces into economic theory a new variable for
processing information—mental models. The new variable is shaped by institutions
through experience and exposure, and activates four mechanisms of social exclusion
discussed in the remainder of the paper.
Barrier 1: Implicit discrimination

Rational choice theory offers two explanations for discrimination: taste-based and
statistical. Taste-based discrimination arises when individuals dislike members of a
group. Becker (1957) argued that employers who discriminate based on social
Hoff and Walsh 5
identity knowingly incur higher costs. Taste-based discrimination would therefore
not survive in perfectly competitive markets.
Statistical discrimination arises when individuals have imperfect information about
an individual and assess his expected productivity based on his membership in a
group (e.g., a race, social class, or gender) (Arrow 1973). Employers may refuse to
hire a member of a group because of a rational determination that the person has
lower expected productivity than job applicants who are members of another group.
A third kind of discrimination, left out of traditional economics, is implicit (uncon-
scious) discrimination (Banaji and Greenwald 1995; Greenwald and Krieger 2006).
Implicit discrimination differs from explicit discrimination in many ways: its sources,
malleability, and effect on behavior. Explicit and implicit discrimination do not emerge
from the same socialization process (Dovidio et al. 1997). Explicit discrimination is
much easier than implicit discrimination to change (Wilson et al. 2000), which is
consistent in that self-reports of explicit racism show a large decline among whites,
whereas racially discriminatory behaviors remain common among them (Dovidio
and Gaertner 2004). Implicit bias predicts important life outcomes. Nosek et al.
(2009) find that cross-country variation in implicit attitudes against women in sci-
ence predict gender-based achievement gaps in eighth-grade science and math. As
we mentioned above and discuss more below, implicit bias by supervisors against mi-
nority staff can directly impair their performance.
Beaman et al. (2009) investigated how having women in leadership positions af-
fected the attitudes and the behavior of constituents. The investigators studied the im-
pact of a policy in India in which the government randomly reserved—in one-third of
the villages—the position of village council leader for women candidates. The study
revealed that the impact was very consequential. The investigators used the Goldberg
paradigm, which is a common way to measure bias; they asked subjects in one group
to evaluate a taped speech by a man, and asked subjects in another group to evaluate
the identical speech made by a woman. In villages that had never had political quotas
for women, both male and female respondents gave the male politician higher ratings
than the woman for effectiveness. But in villages with political quotas for women for
the previous seven years, men evaluated the woman’s speech as just as effective as the
man’s. Discrimination by the measure of the Goldberg paradigm had been removed.
Goldin and Rouse (2000) find evidence of gender bias in hiring for symphony or-
chestras in the United States. Before 1980, none of the five highest-ranked U.S. or-
chestras had more than 12 percent women. Through the 1970s and 1980s, the share
of women hired by the orchestras increased—from about 10 percent in 1970 to about
35 percent in the mid-1990s. During this time, most orchestras introduced screens
that hid the identity and gender of applicants from the hiring panel when they au-
ditioned. Using data from audition records, the investigators found that “blind” au-
ditions increased the probability by 50 percent that a woman would advance from
preliminary rounds. The researchers attribute about 30 percent of the gain in the
number of female musicians in orchestras to the advent of blind auditions.
Crimes by African Americans are understood differently than crimes by whites. For
example, Pager, Western, and Bonikowski (2009) find evidence of the much greater
cost to African American job applicants than to white job applicants of having a
prison record. The investigators recruited African American and whites to apply for
the same set of jobs with similar fictitious résumés, except that one group had résumés
with a prison record and the other group did not. The participants applied in person
for the jobs. African Americans were only half as likely as equally qualified whites to
receive a callback or job offer. Moreover, African American applicants without a crimi-
nal record were no more likely to receive callbacks or job offers than white applicants
with a criminal record. The study describes the experience of the participants (the
“testers”):
In applying at an auto dealership … testers met with very different reactions [by
race]. Joe, the black tester, was informed at the outset that the only available po-
sitions were for people with direct auto sales experience. … When the employer
interviewed Keith, their white ex-felon test partner, he gave him a stern lecture re-
garding his criminal background. The employer warned, “I have no problem with
your conviction, it doesn’t bother me. But if I find out money is missing or you’re
not clean or not showing up on time I have no problem ending the relationship.”
Despite the employer’s concerns, Keith was offered the job on the spot (p. 790).
Salience is a central theme of behavioral economics. If individuals were unbound-

edly rational, all facts about an individual and a situation would be equally accessible.
But for boundedly rational individuals, situations are simplified, and how they are sim-
plified reflects attitudes that can contribute to discrimination. Women are promoted
at lower rates than men in science, technology, engineering, and math (STEM) fields.
Sarsons (2017) shows that an important explanation for the low promotion rate of
women economists is that coauthored research publications by female economists
matter less for tenure than coauthored research publications by male economists. It
is not known how much of this discrimination is implicit. But a recent University of
California, Berkeley senior thesis by Wu (2017) found a way to measure attitudes
towards women in economics that suggests the importance of implicit discrimina-
tion. In an online, anonymous message board that had more than one million posts,
Economics Job Market Rumors, female economists were much more often than male
economists perceived in ways unrelated to their professional roles. The three most
common words used to describe female economists were “hotter,” “lesbian,” and “bb”
(baby). For men, the most common words were “mathematician,” “pricing,” and “ad-
viser.”
Implicit prejudicial attitudes can be self-fulfilling, just as explicit prejudice and dis-
crimination can be (Akerlof 1976). In the empirical study of grocery store cashiers
in France mentioned at the beginning of this essay, workers of African origin were
Hoff and Walsh 7

substantially less productive on the days they were supervised by implicitly biased
managers, where implicit bias was measured by the Implicit Association Test (Glover,
Pallais, Pariente 2017). Bertrand, Chugh, and Mullainathan (2005) argue that IATs
tap unconscious attitudes, which activate a different part of the brain than the parts
that engage in conscious deliberation. When the employees of African origin worked
with unbiased supervisors, they were 9 percent more productive than clerks of non-
African origin; but when they worked with biased supervisors, they had only average
productivity. The authors find evidence that the productivity decline arose because
biased managers avoided interactions with minorities. One reason for the low super-
vision was that the supervisors were worried that they would be accused of bias if
they made an inappropriate remark in their interactions.
A two-stage experiment in Sweden provides additional evidence that implicit bias
contributes to discriminatory behavior (Rooth 2010). In the first stage of the ex-
periment, employers received equivalent applications to advertised jobs from ap-
plicants with Swedish names and from applicants with Arab-Muslim sounding
names. All applicants were represented as male. Rooth contacted the recruiters
to ask if they would participate in tests for explicit and implicit attitudes to-
ward Arab Muslims. The tests demonstrated that employers had strong, explicit
negative attitudes toward Arab-Muslims, though not on the basis of beliefs re-
garding lower productivity: A clear majority of employers (77 percent) stated
that there were no performance differences between the two groups. Implicit at-
titudes predicted the difference in callback rates between applicants with Arab-
Muslim sounding and Swedish names much more reliably than did explicit atti-
tudes. The probability of a callback for Arab-Muslim job applicants declined by
5 percent for each one standard deviation increase in negative implicit association
of Arab-Muslim men.
If prejudice reflects implicit thoughts, not conscious tastes or statistical discrimina-
tion, discriminatory beliefs and attitudes may be sensitive to subtle contextual cues. In
one experiment, individuals were asked which group they preferred—a group of well-
liked African American athletes or a group of disliked white politicians. Respondents
preferred the first group when the context emphasized occupation, but preferred the
second group when it emphasized race (Mitchell, Nosek, and Banaji 2003).
Context influences the salience and valence of categories. Shayo and Zussman
(2011) investigate more than 1,500 judicial decisions in Israeli small claims courts,
where cases are randomly assigned to Arab or Jewish judges. These authors find evi-
dence of judicial in-group bias: a claim is 17 percent to 20 percent more likely to be
accepted if assigned to a judge of the same ethnicity (Arab or Jewish) as the plaintiff.
Consistent with the emphasis in behavioral economics on the effects of salience on
attention, the ethnic bias increases with the population-adjusted number of fatali-
ties in the year preceding the ruling that are from Palestinian politically-motivated
attacks in the vicinity of the court. Terrorism leads Arab judges to favor Arab
plaintiffs and Jewish judges to favor Jewish plaintiffs. Shayo and Zussman conclude
that ethnic conflict, by intensifying ethnic identities, can erode trust in the rule of law:
“There is rather little ethnic ingroup bias in the Israeli courts except during periods in
which political violence intensifies ethnic identification. In other words, by heighten-
ing identification, ethnic conflict can dramatically undermine the proper functioning
of an ostensibly impartial institution like the court system” (2011).
Interventions to Reduce Discrimination
Identification of the impact of implicit discrimination leads immediately to the ques-

tion of whether interventions can reduce it. Evidence suggests that information alone
may not be enough to change beliefs, since individuals tend to resist corrective infor-
mation that calls their priors into question (Nyhan and Reifler 2010; the defensive
reaction is called counter-arguing). However, a longstanding body of work on preju-
dice reduction emphasizes the beneficial impact of interactions between social groups
(Allport 1954; Pettigrew and Tropp 2006).3 Much of this work argues that prejudice
results from a lack of experiential knowledge and understanding of outgroups, and
that it can be offset by interactions between the groups. A recent quasi-natural exper-
iment provides strong supportive evidence (Rao 2018). Many private schools in New
Delhi lease land from the government at heavily subsidized rates. In exchange, the law
requires that the schools admit randomly chosen poor children tuition-free. The law
was not enforced until 2007, when a Delhi High Court decision ordered almost 400
private schools to reserve one-fifth of their seats for students from households with
incomes of less than about $2,000 per year. The schools would be partially compen-
sated by the government. The order applied only to new admissions of students, who
usually enter in pre-school, and required the students to be integrated into the same
classrooms as the non-scholarship students.4
Rao discovered that exposure for four years to poor classmates and, in particular,
regular interaction with them in cooperative groups, changed the social preferences
and behavior of the rich students. When offered opportunities to come to school over
the weekend to support fundraising efforts for a charity that served disadvantaged
children, rich students in cohorts with poor classmates sent 10 percent more volun-
teers than those without poor classmates. In dictator games, rich students who had
poor classmates shared 45 percent more of their endowment, and were much more
likely to split their endowment equally with the other player.5
To measure the impact on discriminatory behavior by the rich students, Rao de-
vised an experiment involving a relay race, with a prize going to the winner. Before
the relay teams were formed, each student ran a sprint individually, which revealed
how fast he ran. Then participants selected their teammates in the relay race. As Rao
explains, “By having participants choose between more athletic poor students and
less athletic rich students, I create a tradeoff between ability and social similarity,”
Hoff and Walsh 9

(2018). In the relay race with the smallest prize (about $0.85), almost one-third of
the rich students without poor classmates discriminated against poor students. Those
with poor classmates discriminated 12 percentage points less.6
Recent research suggests that even a brief interaction between individuals in in-
groups and out-groups can sometimes shift tastes and beliefs. Evidence that bears this
out is an experiment on attitudes towards transgender people, who face high levels of
prejudice and discrimination in the United States. In a randomized field experiment,
voters were canvassed by activists for transgender rights. The canvassers, some of
whom were transgender, told voters that they might be asked to vote on whether to re-
peal legal protections for transgender people. Voters were asked to discuss their views
and were shown a video presenting the arguments on each side. After the video, the
canvassers encouraged voters to engage in “perspective taking”—discussing an oc-
casion when they felt judged negatively for being different and considering whether
their own experience offered a window into the experiences of transgender people.
The brief interaction had a large effect. As compared with a control group (voters
who were canvassed on recycling issues), the intervention increased positive attitudes
toward transgender people by 10 percentage points. To put this in perspective, the im-
pact is larger than the increase in positive attitudes toward gay men and lesbians in
the United States between 1998 and 2012 (Broockman and Kalla 2016).
There is also evidence that a brief intervention can change behavior in a high-
stakes interaction. Disciplinary problems of students are a strong predictor of neg-
ative life outcomes; school sanctions early in life can set off a long-lasting negative
trajectory (Rocque and Paternoster 2011). In an experiment in five racially diverse
middle schools in California, a brief intervention encouraged math teachers to adopt
an empathetic instead of a punitive mindset with regard to discipline (Okonofua and
others 2016). The teachers were told that the purpose was to review “common but
sometimes neglected wisdom about teaching and to collect their perspectives as ex-
perienced teachers on how best to handle difficult interactions with students, es-
pecially disciplinary encounters.” The teachers read an article about reasons why
students misbehave (e.g., social and biological changes during adolescence, worries,
stresses, and social anxiety), which discouraged teachers from labeling students as
troublemakers. The article encouraged teachers to place value on students’ experi-
ences and to develop and sustain positive relationships with the students. This was
reinforced with stories from students. The intervention neither discouraged disci-
plinary actions nor encouraged the view that students’ perspectives were necessarily
reasonable. The intervention reduced suspensions from school. Students of teachers
who received the treatment were half as likely to be suspended over the academic
year.
An alternative way to change behavior is to increase self-awareness. Publicizing
the extent of implicit bias may alert people to discriminatory actions and thereby re-
duce, or even eliminate, the discrimination. Price and Wolfers (2007) reported that
fouls in NBA games from 1991 to 2002 were more likely to be called against African
Americans when the referee team was made up only of whites, and were more likely
to be called against whites when the referee team was made up only of African Amer-
icans. The New York Times covered the study on its front page. Other newspapers and
TV stations, such as ESPN, covered it as well. Six years later, Pope, Price, and Wolfers
(2013) showed that the discovery and publicity ended the discriminatory behavior of
the referees. The new study compared the period of 2002 to 2007 (before the pub-
lication of the earlier study) to the period of 2007 to 2010 (after the publication).
In-group bias persisted until 2007, but stopped from 2007 to 2010. The researchers
explored whether the change occurred due to a shake-up in the line-up of referees, but
found that of the 66 referees who had officiated at least 100 games in from 2003 to
2006, 55 officiated at least 100 games from 2007 to 2010. The researchers also con-
sidered whether the NBA changed the racial makeup of the teams, but found that the
fraction of mixed-race referee teams actually decreased. The researchers also spoke
to the NBA league administrators, who indicated that no policies had changed in re-
sponse to the paper. Thus, no observed change in structural conditions, but only a
change in awareness, seems to explain the end of discriminatory refereeing.
It is sometimes possible to make small changes in a decision-making environment
to activate scripts that rely less on stereotypes and thereby reduce discrimination.
Bohnet, Geen, and Bazerman (2016) constructed a game in which participants had
to hire employees for mathematical or verbal tasks. Participants were paid based on
the performance of the person they hired. Participants were given the following in-
formation about a candidate: (a) performance on a prior test, (b) gender, (c) his or
her identity as an American student from the Boston area, and (d) the average per-
formance of the pool from which the candidate was drawn. Participants could hire
either someone offered to them or a randomly selected person from the pool. There
were two experimental conditions. In one condition, participants were offered only
one candidate. In the other condition, participants were offered a male candidate and
a female candidate. All the offered candidates had on a prior test performed at av-
erage or slightly below average level of the pool from which they were drawn. IATs
show that people have implicit beliefs that men have greater math skills than women,
and that women have greater verbal skills than men (Nosek, Banaji, and Greenwald
2002; Plante, Theoret, and Favreau 2009). Consistent with this finding, the partici-
pants were more likely to choose the offered male candidate for a math task and the
offered female candidate for a verbal task. When evaluated on their own, stereotype-
advantaged individuals were chosen 66 percent of the time. However, the stereotype
advantage was present only when participants were deciding on a single offered can-
didate and not present when they had a choice between a man and a woman, whom
they could compare to each other.7 A fine adjustment to the hiring process eliminated
gender discrimination.
Hoff and Walsh 11

The examples presented in this section reduced prejudice for reasons that may be
heavily contingent on context. The findings may not be parsimoniously applicable
to new settings. They may require adaptation to work in new contexts, or they may
not work at all. Research in this area is growing rapidly to identify interventions that
reduce discrimination.
Barrier 2: Self-stereotyping and Self-censorship
Up to now, we have discussed stereotyping of others. But individuals also stereotype
themselves. The stigma of social exclusion can be so profound as to “get into people’s
heads” and degrade their self-concept (Goffman 1963).
To function well, people need to understand themselves in terms that sustain their
sense of personal integrity and value and that accurately reflect their abilities. Many
institutions are designed to support the psychological and social needs of dominant
groups by providing role models, narratives, and rituals to embolden individuals to
make the effort needed to succeed. Groups at the margins of society, on the other
hand, often must expend extraordinary effort to negotiate their place in environments
not made for them or hostile to them. “Like a distracting alarm, psychological threat
can also consume mental resources that could otherwise be marshaled for better per-
formance and problem solving” (Cohen and Sherman 2014, p. 335). Individuals may
spend time assessing whether they belong, whether they are wanted, whether they
are good enough, whether they are worthy (Walton and Cohen 2007). For a given
situation, two individuals—one in an in-group, the other in an out-group—may be
engaging in very different amounts of mental energy.
Even small pressures on mental resources can reduce the ability to self-regulate.
A trivial example demonstrates the point. Shiv and Fedorikhin (1999) asked a group
of students to memorize a number and recall it a few minutes later in another room.
They randomly assigned the students to two groups. One group was given a seven-
digit number to memorize. The other group was given a two-digit number. When the
students left the room, they had a choice of snack as reward for their participation—a
bowl of fruit salad or a piece of chocolate cake. Compared to those asked to mem-
orize the two-digit number, those asked to memorize the seven-digit number were
50 percent more likely to choose the cake, the less healthful choice—63 percent ver-
sus 42 percent
How and when does a negative stereotype become a cognitive tax? The cues in a
situation, the individual’s perception of the situation, and the relevance attributed
to social identity in that situation all influence whether or not the stereotype is ac-
tivated (Okamura 1981). For some individuals, a negative stereotype may be chron-
ically activated. Pioneering studies on the effect of priming social identity find that
merely checking a box to indicate race before taking an aptitude test lowers the
performance of African American students, but not of white students (Steele and
Aronson 1995, 1998). The “race prime” appears to raise the consciousness of
negative stereotypes among African Americans. Steele and Aronson (1998) call this
effect “stereotype threat”: “Participants who experience stereotype threat spend more
time doing fewer items less accurately.”
Stereotype threat has been documented in many contexts. In India, individuals
are born into castes, which in each locality are ranked. High-caste individuals are
traditionally considered socially and intellectually superior to low-caste individuals
(called Scheduled Castes). While discrimination against low caste members is illegal,
low-caste children nonetheless encounter the traditional order of caste and untouch-
ability in the fables they learn and often in the continued insults, discrimination, and
atrocities against upwardly mobile members of low castes. Hoff and Pandey (2006,
2014) assessed the effect of making caste identity public, and of caste segregation on
the performance of junior high school boys in rural north India. Caste segregation
is a mark of the civic privileges of the high castes, and the social exclusion and
inferiority of the low castes (Jodhka 2002). The participants were asked to solve
mazes and paid for each maze they solved. Participants were randomly assigned to
one of three conditions: (a) anonymous, (b) caste revealed in mixed-caste groups,
and (c) caste revealed in groups segregated by caste status (high or low). In the first
condition, three high-caste and three low-caste boys were placed in a session and
their identity and caste were not made public in the session. Since, in general, the
children came from six different villages, their caste would not be known to the
other children in the session. In the second condition, three high-caste and three
low-caste boys were placed in a session and their identity and caste were made public
at the beginning of the session. The third condition was the same as the second
one, except that the six boys in a session were all from high castes or all from low
castes.
The anonymous condition showed that low-caste boys solved mazes just as well
as high-caste boys. However, publicly revealing caste in mixed-caste groups created
a 23 percent caste gap in total mazes solved in favor of the high castes, controlling
for other individual variables. A possible explanation is that the boys felt “I can’t or
don’t dare to excel.” In the third condition, segregation depressed the performance of
both high-caste and low-caste boys. If segregation evokes a sense of entitlement in
the high caste, the high-caste boys may have felt, “Why try?”
The experiment to test the effect on maze-solving ability of making a stigmatized
identity salient was replicated in Beijing, China, although in this experiment the
identity treatment was stronger: in addition to revealing children’s social identity,
students completed a pre-experiment survey that asked questions about their social
identity and about the characteristics of groups with their own and other social iden-
tities (Afridi, Li, and Ren 2015). The subjects were elementary school children aged
8 to 12 drawn from two social categories: (a) households classified as urban Beijing
households, a privileged category, and (b) households classified as rural non-Beijing, a
disadvantaged category in Beijing. The household registration system in China,
Hoff and Walsh 13

known as hukou, classifies citizens based on the birthplace of their parents or grand-
parents, and favors those categorized as local urban residents in housing, jobs, access
to schools, and public benefits. Unlike categories of gender, class, and caste, hukou is a
transparent man-made creation. The experimental findings show that priming hukou
shifts performance in ways that mirror the way the groups are ranked. In this sense,
the social identity “makes up people,” and a group’s alleged inferiority becomes “an
equilibrium fiction” (Hoff and Stiglitz 2010).
Most people have difficulty judging their own ability. Coffman (2014) implemented
an experiment with U.S. college students that reveals that a person’s judgment of his
ability in a given domain depends on the interaction between his gender and the gen-
der stereotype associated with the domain. In the experiment, which minimized dis-
crimination and fear of discrimination, female participants under-contributed their
ideas in male-typed domains and vice versa; that is, they self-censored. If women and
men are less likely to contribute their ideas, this will hinder their advancement; it is a
self-administered kind of social exclusion. The variable that explains it is the mental
model of gender.
A related mechanism that can lead to the replication of inequality after structural
and institutional barriers have been removed is coordination on unequally rewarded
tasks. In academic departments, individuals have some choice over how much of their
time to spend on non-promotable tasks, for example, attending committee meetings,
evaluating applicants, and advising undergraduates. There is extensive evidence in
academia and industry that women spend more time on non-promotable tasks than
men (see references in Babcock et al. 2017). While many factors could explain this—
for example, gender differences in preferences and abilities—it could also be driven
by shared expectations regarding the appropriate behavior of women and men. To
investigate this, Babcock et al. (2017) ran controlled experiments. They examined
how gender affects the allocation between men and women of a relatively poorly paid
task. In each of ten rounds, participants—all seated in one large room and each with
a computer—were randomly divided into groups of three persons. Members of the
group had to make one decision—to volunteer, or not, to be the poorly-rewarded per-
son in the group. Each round could last at most two minutes. To volunteer, a person
clicked on his or her screen. As soon as a group member clicked, or two minutes had
elapsed, the round ended. The incentives were as follows: each player in a group got
$1 if nobody in the group volunteered to be the poorly rewarded person. If somebody
volunteered, the volunteer got $1.25 and the other two people in the group each got
$2.00.
When the participants in the room were roughly half men and half women, so
that individuals knew that their group was very likely to be mixed gender, women
volunteered twice as often as men. When the participants in the room were all men
or all women, so that individuals knew that their group was single gender, men and
women were equally likely to volunteer for the poorly-rewarded task. The results are
consistent with the hypothesis that mental models of gender have a large effect on
behavior, and tend to replicate historical inequalities.
In a follow-up experiment, individuals played a similar game in which there was a
fourth member of each group. His or her only role was to try to resolve the coordina-
tion problem by asking one member to volunteer to be the poorly-rewarded person.
Women received more requests than men by a factor of 2.50. The gap increased as
the game was repeated over ten rounds. When asked to volunteer, women were 49
percent more likely than men to agree. The results suggest that shared expectations
based on traditionally unequal gender roles replicate the inequality when individu-
als coordinate anonymously in novel situations on unequally rewarded tasks. The
traditional gender roles carry over to the novel situations even though there is no
structural basis for the carry-over.
Another area in which gender roles affect behavior in ways that replicate histori-
cally imposed inequalities is in salary negotiation. Lab and field evidence suggests that
men are more likely to ask for higher compensation than women (Small et al. 2007).
But seemingly minor situational factors loom large in individuals’ decision-making.
In a field experiment, List and Leibbrandt (2014) replicate the result in Small et al.:
they find that when salaries are not explicitly made negotiable, women are 23 percent
less likely to negotiate for higher salaries; but when it is clear that salaries are nego-
tiable, women are 8 percent more likely than men to negotiate for higher salaries.
Interventions to Weaken the Influence of Negative Stereotypes on Self-assessment
Minor interventions can insulate socially excluded groups from the “threat in the air”
created by social stigma. The next five paragraphs discuss experiments with disadvan-
taged students in the U.S. Then we discuss an intervention in India that reduced the
legitimacy of domestic violence.
Experiments in the United States show that interventions that inculcate feelings
of belonging can improve academic performance among non-traditional college stu-
dents (Yeager et al. 2016). In one experiment, disadvantaged high school students
who had been admitted to two- and four-year colleges were invited to participate in
an online module designed to dispel the belief that disadvantaged students are the
only group that has difficulty in college or the only ones who question whether they
belong in college. One year later, 45 percent of the students who had participated in
the intervention were enrolled full-time in school, compared to 32 percent of the con-
trol group. A similar experiment, in which participants had already entered college,
reduced the enrollment gap between disadvantaged and advantaged students by 40
percent.
An intervention that has been tested in multiple contexts is a values affirmation
exercise. In one experiment, an essay assignment was given to students through
their normal coursework two times during the school year (Cohen et al. 2006). The
Hoff and Walsh 15

assignments asked students to think about their most important value. It took 10
to 15 minutes to complete an essay on this topic. Students knew their teachers
would read their essays, but the teachers did not know the identity of the students in
the treatment. The treatment improved the grade point average of socially excluded
groups (African Americans and Latino Americans) in academic courses, and reduced
by 50 percent the proportion of African Americans receiving a D or an F in their
first term. The effects persisted over the two years in which participants’ grade point
averages were tracked. African American students’ grades improved by 0.24 grade
points. Among low-achieving African Americans students, performance improved
even more: grade point averages increased by 0.41 points, and the rate of remedi-
ation or grade repetition was less than a third of that of the control group (Cohen et
al. 2009)
Another way to help socially excluded groups improve performance is to frame the
idea of intelligence as a malleable trait that grows in response to hard work, rather
than as a fixed trait (Hong et al. 1999; Dweck 2006; Nussbaum and Dweck 2008).
Disadvantaged groups may be more likely to believe that intelligence is fixed rather
than malleable (Claro, Paunesku, and Dweck 2016). In a seminal study, Aronson,
Fried, and Good (2002) tested the impact of fostering a “growth mindset” among
African American college students.8 Students were taught the theory of malleable in-
telligence in three one-hour lab sessions and were encouraged to explain the ideas in a
letter to an at-risk middle school student. The intervention increased the participants’
belief that intelligence was malleable, their enjoyment of academics, and their belief
that academics are important; the intervention increased the participants’ semester
grade point average from 3.05 to 3.32.
Many interventions to counter the belief that intelligence is a fixed trait have been
effective in small or lab settings, but can such interventions be effective at scale and for
a heterogeneous population? Recent and ongoing work investigates this. Paunesku
et al. (2015) designed and delivered two online 45-minute-videotapes to over 1,500
students in 13 geographically diverse U.S. high schools. One video communicated the
idea of malleable intelligence and growth mindset. The other video encouraged stu-
dents to reflect on how working hard at school can help the students accomplish
meaningful goals. One-third of participants were at risk of dropping out of high
school.
The intervention was a success. Compared to a control group, both treatments
raised students’ grade point averages in core academic courses. The fraction of at-risk
students who satisfactorily completed core courses was 6.4 percentage points higher
for the treatment group than for the control. The results show that interventions can
be delivered at low-cost and at scale. The World Bank’s behavioral science unit, the
Mind, Behavior, and Development Unit (eMBeD), has delivered paper-based growth
mindset interventions to approximately 40,000 students in Peru and 200,000 stu-
dents in Indonesia. The preliminary results from Peru are promising.
Narratives are a source of shared mental representations. Some narratives enable
people to see situations in a way that spurs activity. Other narratives constrain agency
by representing a person as unable to influence outcomes outside a narrow domain.
Famous examples of positive narratives that have shaped many Americans’ views of
poverty are the rags-to-riches novels by Horatio Alger: no matter how dire the hero’s
straits at the beginning, every hero in Alger’s stories escapes poverty by dint of effort,
ability, and inner strength.
A Theater for Development program, Jana Sanskriti, active in villages in West
Bengal, India since 1985, engages in fieldwork in which villagers describe incidents of
domestic violence and other problems that they face. The artistic director then writes
plays that are performed in the villages to dramatize the problems people experience
in their daily lives. As Augusto Boal, the Brazilian writer and politician who created
Theater for Development, explains,
When the skit is over, the participants are asked if they agree with the solution
presented. At least some will say no. At this point it is explained that the play will
be performed once more, exactly as it was the first time. But now any participant
in the audience has the right to replace any actor and lead the action in the direc-
tion that seems to him most appropriate… The other actors have to face the newly
created situation, responding instantly to all the possibilities that it may present…
Boal (1973).
The goal of Jana Sanskriti is to enable villagers to change their shared representa-
tions, for example, of domestic violence, and collectively rehearse social change. To
evaluate the impact, Hoff, Jalan, and Santra (in progress) surveyed random samples
of registered voters in villages where the plays had been performed and in matched
villages where plays had never been performed. The study finds that exposure of a vil-
lage to the plays reduced to less than 5 percent the fraction of both men and women
who thought domestic violence was legitimate. Exposure reduced the percentage of
households in which domestic violence had recently occurred from 32 percent to 26
percent. By providing an entry point for communities to collectively contest traditional
social norms, the theater program may expand individuals’ cognitive toolkit for inter-
preting domestic relationships. In the new mental models, domestic violence is per-
ceived as cruel and illegitimate.
Barrier 3: The Challenge of Adapting to Two Worlds

The title of Kahneman’s (2011) book, Thinking, Fast and Slow, captures a central tenet
in behavioral economics and a major theoretical development in the understanding
of human behavior: individuals have a dual cognitive process. “Fast” thinking is in-
tuitive and automatic, and “slow” thinking is deliberate and effortful. Fast thinking
is generally well-adapted to environments that one knows well: “Time and energy
Hoff and Walsh 17

are saved, rumination and doubt are reduced, and nothing important is lost” (Ross
and Nisbett 1991; see also Gigerenzer and others 1999). But in less familiar envi-
ronments, fast thinking may lead to systematic mistakes. Economically and racially
segregated neighborhoods put disadvantaged individuals in a difficult position: they
do not access as young children the more privileged settings in which they can learn
the norms and rules of thumb needed to thrive in such environments.
When the U.S. government offered people living in a poor neighborhood vouchers
to move to a higher-income neighborhood, young children’s earnings later in life im-
proved by $3,477 on average per year, an estimated $302,000 in a lifetime (Chetty,
Hendren, and Katz 2016). A likely benefit of moving to better neighborhoods includes
access to the social and cultural capital to function well in middle-income environ-
ments (Wilson 1987; Sampson 2012).
Individuals in out-groups must often navigate from a young age two very differ-
ent cultural worlds—their home environment, with its epistemology and norms, and
the world of the in-group, with different epistemology and norms. Children who live
in areas plagued by high rates of crime must learn one set of rules of thumb to
survive in their neighborhood and another set to thrive in school. School environ-
ments are heavily regulated by formal authority and norms of civility. In contrast, in
high-crime neighborhoods, power and authority are fluid and negotiable. Anderson
(1999) writes that “one of the most salient features of urban life in the minds of many
people today is the relative prevalence of violence.” For people living in neighbor-
hoods that lack institutions to prevent crime, conduct is typically regulated by the
threat of violence—the “code of the street.” Coates (2015) wrote the following of his
time growing up in Baltimore:
To survive the neighborhoods and shield my body, I learned another language con-
sisting of a basic complement of head nods and handshakes. I memorized a list
of prohibited blocks. I learned the smell and the feel of fighting weather. And I
learned that “Shorty, can I see your bike” was never a sincere question, and “Yo,
you was messing with my cousin” was neither an earnest accusation nor a misun-
derstanding of the facts. These were the summonses that you answered with your
left foot forward, your right foot back, your hands guarding your face, one slightly
lower than the other, cocked like a hammer.
Particularly for males who live in dangerous neighborhoods, there is generally no

set of automatic responses to the assertion of authority that they can apply success-
fully both in their neighborhoods and in the school or workplace. An automatic re-
sponse of compliance would endanger them in their neighborhoods, whereas an au-
tomatic response of non-compliance in school could lead to suspension, expulsion
or termination. Middle-class youth normally do not face this problem. For them, the
appropriate response to authority is the same in the home and school environment.
Interventions to Support Adaptive Strategies of Disadvantaged Groups
Traditional economics takes the person’s preferences as fixed. The standard policy
prescription to deter disorder and violence is punishment. To encourage individuals
to invest in their human capital, the standard policy prescription is to provide them
information on the benefits or greater incentives are advised. In contrast, behavioral
economics recognizes that the person can revise his mental models, behavior, and per-
formance if given adequate social and psychological supports. This section discusses
interventions that have increased individuals’ life skills and raised their aspirations.
One of the simplest ways for socially excluded groups to learn the rules of thumb
necessary to succeed in more privileged settings is mentoring and coaching. Bettinger
and Baker (2011) evaluated a mentoring and coaching service for non-traditional
U.S. college students. Coaches worked with students to help them clarify their aspi-
rations, connect their daily activities to long-term plans, and build skills such as time
management and self-advocacy. Coached students were 14 percent more likely to per-
sist in school after 24 months and four percentage points more likely to complete their
degree within four years of receiving the treatment.
All-encompassing interventions are not always needed to change behavior in very
positive ways. In some cases, rules of thumb lead to poor outcomes and can be
changed. In South Africa, women who seek jobs generally do not ask for reference
letters from former employers. When they do ask them to send letters, callback rates
increase by 89 percent (Abel et al. 2017). One might suppose that this is because more
capable women seek letters; their superior abilities, not the letters, are the reason for
the higher callback rates. But this was not the explanation. A randomized controlled
trial (RCT) that encouraged women in the treatment group to seek and use reference
letters doubled their employment rates. In contrast, getting reference letters had no
effect in the case of male applicants. Data collected three months after the interven-
tion shows that the intervention closed the gender gap in job-search success.
Another strategy to lead people to adopt better rules of thumb in a particular do-
main is to encourage goal-setting. Deliberating and focusing on specific, challenging
goals stimulates goal-directed behavior (Locke and Latham 1990). Goal-setting fo-
cuses attention on aspired states. It makes salient the losses incurred if one does not
reach one’s goal, the relationship between steps necessary to achieve the goal, and
the goal itself. An experiment in Canada recruited 85 low-performing university stu-
dents and randomly assigned half of them to an intervention (Morisano et al. 2010).
Students in the treatment group were invited to write down their aspirations, values,
role models, priorities, and the ways that achieving their goals would affect the lives
of other people. The students in the control group were asked to write about earlier
positive experiences. Figure 1 shows that the students who participated in the goal-
setting intervention had grades in the next semester almost half a point higher (on a
Hoff and Walsh 19

Figure 1. Low-performing Students Who Were Treated in the Goal-setting Intervention Made
Gains in Their Grade Point Average Compared to the Control Group
Note: Morisano et al. (2010)
scale of four) than the control group. A related experiment with similar results can
be found in Schippers et al. 2015.
One variant of the goal-setting approach is called “WOOP” (wish, outcome, ob-
stacle, plan). This approach combines mental contrasting with detailed implementa-
tion intentions (Oettingen 2014). Mental contrasting entails visualizing your goals,
the reasons they are important to you and the people around you, and considering
the obstacles to achieving them. Implementation intentions involve making detailed
“if. . ., then. . .,” plans to overcome obstacles (Gollwitzer 1999). Oettingen finds that
contrasting their goals with the barriers to achieving them enhances motivation.
Duckworth et al. (2013) implemented an intervention based on WOOP to eleven-
year old children from disadvantaged backgrounds in the United States. The students
were given a worksheet packet and asked to write down their most important wish
or goal related to school work—“something that is challenging, but that you can
achieve within the next few weeks or months,” the instructor explained. The children
were also asked to write down “the one best outcome, the one best thing of fulfilling
your wish or reaching your goal.” The children were given time to imagine the out-
come they had written about, and then randomly assigned to one of two groups. In
the treatment group, students were asked to imagine an obstacle they might face in
achieving their wish and to create “if. . ., then. . .,” plans to overcome it. In the con-
trol group, students were asked to imagine a second positive outcome. The treatment
group subsequently performed better than the control group: they had higher report
card grades, higher attendance rates, and better conduct (Duckworth et al. 2013).
We next return to the central example in our discussion of barrier 3—automatic
aggression in response to assertions of authority. Heller et al. (2017) evaluated RCTs
of cognitive behavioral therapy programs to reduce automatic aggressive responses
by disadvantaged male youths in the Chicago area. Two of the treatments were a pro-
gram called “Becoming a Man.” The third treatment was a program in a Juvenile Tem-
porary Detention Center.
A simple activity illustrates how the Becoming a Man program worked. The activity
leaders used a provocative exercise to show how participants automatically followed
one strategy rather than taking a moment to weigh their options. Activity leaders
broke groups of participants into pairs and gave one person in each pair a ball. The
other person was instructed to get the ball from him. He was given 30 seconds to
do so. The automatic response of almost all the participants was to use force to take
the ball. After the exercise, the activity leader pointed out that the participants could
have asked for the ball. When prompted for an explanation as to why they had not
done that, they usually responded that their partner would not have complied. The
activity leader then asked the partner what he would have done if asked, and most
partners said that they would have given the ball.
The participants in Becoming a Man had an average GPA of 2.0 (out of 4.0), had
typically missed six to eight weeks of school in the year, and in many cases had a his-
tory of arrests. The intervention reduced participants’ interactions with the criminal
justice system. The first intervention, delivered to boys in seventh to tenth grade, re-
duced violent crime by 44 percent and non-violent crime by 36 percent.9 Program
participants also became more engaged in school, which the authors forecast could
translate into increases in graduation rates of about 7 percent to 22 percent.10 The
second intervention, delivered to boys in ninth and tenth grade, reduced arrests by
31 percent. The third intervention, delivered at the Juvenile Temporary Detention
Center, reduced by 16 percentage points the re-admittance rate to the detention cen-
ter within 18 months of release.
Cognitive therapy interventions have also been tested in Sierra Leone and Liberia.
In Sierra Leone, Betancourt et al. (2014) evaluated a youth readiness intervention
that delivered psychosocial supports to war-affected youth. The goal was to help them
regulate their emotions and improve their problem-solving skills. The treatment pro-
vided all participants with an education subsidy and randomly allocated the psy-
chosocial support treatment. Students who received the psychosocial supports per-
formed better and were more likely to stay in school.
In Liberia, Blattman, Jamison, and Sheridan (2017) partnered with a local organi-
zation to provide group-based therapy and/or $200 cash (about three months’ wages)
to almost 1,000 criminally-engaged men. Participants were randomly divided into
four groups: one group received only the cash grant; a second group received eight
weeks of therapy designed to foster self-regulation, patience, and a non-criminal iden-
tity; a third group received both the grant and therapy; and the control received nei-
ther. Those who had received only the cash transfer made no changes in criminal
behavior or self-regulation. Among those who received only the therapy, violent and
criminal behavior declined—in the short run, the individuals were 55 percent less
likely to carry a weapon and 47 percent less likely to sell drugs. The effects were
longer-lasting and stronger among those who had received the therapy followed by
Hoff and Walsh 21

the cash grant. The participants in this third group became less impulsive, had higher
self-esteem, and were less likely to steal for at least a year after the intervention.
The studies discussed in this section show that short-term interventions can in-
duce changes in behavior with long-run effects on well-being. One set of interven-
tions shifted individuals from a “fixed” to a “growth” mindset of intelligence and
helped them form goals, strategies to achieve them, and the emotional skills to fol-
low through. Another set of interventions gave individuals who grew up in unsafe
neighborhoods the mental tools to adapt to a non-violent environment and build pro-
ductive lives.
Barrier 4: “Adaptive Preferences”

Traditional economics, as we have emphasized throughout, assumes fixed prefer-
ences, whereas Strand 2 of behavioral economics does not. In the past two decades,
lab and field and “as-if-random” natural experiments (in which we include the work
on gender and the plough) have provided evidence that is difficult to explain except
through the impact of experience and exposure on preferences. Taking account of
the malleability of preferences sheds light on a particularly perverse driver of social
exclusion: the “adaptive preferences” of the oppressed (Nussbaum 2001). That is, a
subjugated group may come to see its subjugation as natural, normative, or even pre-
ferred. Bourdieu (2000) writes that “the realistic, even resigned or fatalistic, dispo-
sitions which lead members of the dominated classes to put up with objective condi-
tions that would be judged intolerable or revolting by agents otherwise disposed. . .help
to reproduce the conditions of oppression” (emphasis added). Fatalistic attitudes and
beliefs can be difficult to tackle in part because they call into question whether adults
understand their own preferences. Duflo (2012) describes “hope” as a form of capa-
bility.
Examples of “adaptive preferences” in varying degrees of intensity exist in many
contexts. For example, Guyon and Huillery (2014) find evidence of a large so-
cial class gap in aspirations of 14-year old students in the area of Paris. In many
countries, as shown in figure 2, a high proportion of women report that a hus-
band is justified in beating his wife for refusing sex. One reason that preferences
adapt is that existing institutions serve as reference points—the baseline relative to
which people imagine alternative realities. People have limited ability to imagine
counterfactuals.
In parts of rural India, most girls leave school early, marry, and have children at
a young age. The explanation in traditional economics would be that the behaviors
reflect the preferred outcomes of the girls or their families given their opportunity
sets. However, a recent RCT points also to the role of limited opportunities to imag-
ine alternative ways of life. In a large-scale field experiment, Jensen (2012) provided
three years of recruiting services to help women in remote villages in India get jobs in
Figure 2. The Legitimacy of Wife-beating
Note: Based on data from DHS Statcompiler (http://www.statcompiler.com).
the business process outsourcing industry. Although, on average, only three women
per village were hired, young women in the treatment villages became less likely to
be married or have children, and more likely to work or continue their education.
The proportion of young women aged 15 to 21 who were married dropped from
71 percent to 66 percent, and the proportion with children dropped from 43 percent
to 37 percent. In addition, the treatment closed 30 percent of the gap in body mass in-
dex between girls in the villages and girls in the wealthiest families in New Delhi. Was
this simply a reflection of pre-existing preferences responding to new opportunities?
Perhaps. But an equally plausible interpretation is that there was also a change in
preferences: exposure to local village women who got good jobs helped young women
imagine better lives for themselves.
One way for adaptive preferences to emerge is through a perverse trusting relation-
ship that oppressed individuals may develop with their oppressors. This is sometimes
called the Stockholm syndrome (Namnyak et al. 2008). The term emerged from a
dramatic event in Sweden. On a summer morning in 1973, a prison-escapee entered
a bank with a submachine gun and shot a police officer. In the failed bank robbery,
he took four hostages and demanded that his prison mate be released from prison
and join him. The government acquiesced. The two men barricaded themselves in the
bank, with the hostages locked in the bank vault. Astonishingly, the hostages began
to develop a bond with their captors and resisted cooperation with the police.
Hoff and Walsh 23

Examples related to the Stockholm Syndrome exist at the level of a whole society.
In Sierra Leone, villages are ruled by “paramount chiefs” from families originally rec-
ognized by British colonial authorities. The number of ruling families varies across
villages due to accidents of history. Villages with fewer ruling families have worse gov-
ernance, child health, educational attainment, and incomes (Acemoglu, Reed, and
Robinson 2014). One might expect that villages with poorer development outcomes
would have less respect for authority and be less satisfied with the ruling families. But
this is not the case. The researchers found that villages with fewer ruling families re-
ported higher respect for authority.
In the Indian state of Maharashtra, local government is by a variety of objective
measures more oppressive in villages in which the high castes own most of the land.
To increase the extent to which the landless depend on them, the high castes block
many national pro-poor programs. Yet the perceived legitimacy of village govern-
ment is higher in high-caste-dominated villages: low-caste residents are 14 percent
more likely to report trusting the landholders in the high-caste-dominated villages
(Anderson, Francois, and Kotwal 2015).
Role Models
What can change dysfunctional “adaptive preferences”? By observing others, indi-

viduals may develop aspirations and more positively assess their prospect of achiev-
ing them (Bandura 1986, 1997). Role models who achieve success help individuals
imagine new life paths and may boost self-efficacy beliefs.
A natural experiment in India shows the impact of creating role models by secur-
ing places for women in positions of power. As discussed above, for the position of vil-
lage council leader (Pradhan), in the 1990s the government of India mandated quo-
tas for women in a randomly selected one-third of all villages. Exposure to women
Pradhans changed beliefs in in at least four ways that reduced the social exclusion
of women (Beaman et al. 2009; Beaman et al. 2012). (1) The experience of living
under a woman Pradhan erased the prejudice, on average, of male villagers against
women leaders by many measures—the evaluation of political speeches (discussed
above), an IAT, and the assessment of the quality of actual village Pradhans. (2) In
villages that had women leaders, parents’ aspirations for their teenage daughters—
and teenage girls’ aspirations for themselves—were higher; and girls have gone to
school somewhat longer and done somewhat fewer hours of housework. (3) After
the quota program ended in a village, women had run for political office in higher
percentages and in many cases won (see figure 3). (4) The greater presence of fe-
male political representatives produced an unexpected change in women’s reporting
of crimes against women, and in the willingness of the police to record the reports
(Iyer et al. 2012). This occurred even though Pradhans have no jurisdiction over the
police. The increased reporting by female victims of crimes appears instead to reflect
Figure 3. The Fraction of Women Who Won Office in Free Elections
Note: Figure from World Bank (2015), data from Beaman et al. (2009).
a change in their perception of the costs—psychic and otherwise—of reporting the

crimes.
Another way to change the “adaptive preferences” of disadvantaged groups is
through documentaries of people who escaped poverty, and soap operas that expose
viewers to new ways to build their lives. A field experiment conducted by Bernard et
al. (2014) in Ethiopia examined the impact of showing a one-hour video with profiles
of four Ethiopian villages that had escaped poverty or entered the middle class. The
team randomly selected about 60 villages for the treatment and about 60 villages for
the control group. The locations were remote—the researchers reached them by 4x4
vehicles or camels. Only 10 percent of the local populations watched TV at least once
a week. The intervention raised aspirations and expectations, and the effect persisted.
Five years later, aspirations were still significantly higher in the treatment than in the
control group. The intervention also increased future-oriented behavior. School en-
rollment of children was 17 percent larger than in the control group six months after
the intervention.
There is evidence that exposure to unfamiliar outcomes and social patterns even
in fiction can give individuals new role models. The rise and popularity of soap operas
in Brazil depicting agentic women with few or no children was a statistically signifi-
cant factor in the recent decline in Brazil’s fertility rate. By taking advantage of the
plausibly exogenous spread of television across Brazilian municipalities, La Ferrara,
Hoff and Walsh 25
Chong, and Duryea (2012) showed that a fertility decline was caused by the soap op-
eras in the year that a municipality first received the emissions. The decline was most
pronounced for women who were close in age to a protagonist of one of the soap op-
eras, which is consistent with the idea that the characters were role models. To put
this effect in a comparative perspective, the impact was similar to the effect of two ad-
ditional years in women’s education. Fertility fell by 11 percent of the mean among
women aged 35 to 44. Organizations such as the BBC Media Action have brought
this approach to scale across the world, tackling problems such as ethnic conflict,
poor health, and oppressive gender norms.
Conclusion
A central theme of this essay is that institutions create concepts, categories, social
identities, and other mental models that can persist long after the institutions are
abolished. The mental models influence how boundedly rational people process infor-
mation and what they want and aspire to become. Reforming or abolishing an institu-
tion to reduce social exclusion may not change the mental models that the institution
has advanced. In that case, social exclusion will persist. The realization of equal sub-
stantive opportunity may require interventions that target socio-psychological barri-
ers to social inclusion and upward mobility.
The interpretation of the causes of social exclusion in traditional economics con-
trasts sharply with the perspective in behavioral economics. If a social group remains
at the bottom of the social ladder long after procedural equality of opportunity has
been established, the implication in traditional economics would be that the group
has fixed characteristics that impede upward mobility, or that network externalities
keep it from rising: the group will move up in socio-economic status only if an event
makes possible a coordinated change in behavior.
In contrast, behavioral economics shows that social exclusion is caused not only
by structural and institutional barriers, but also by socio-psychological factors and
further, that interventions can offset them. A stigmatized ascriptive identity affects
its members in many ways besides procedural barriers to opportunity and explicit
animus. Socially excluded groups face implicit bias. Negative stereotypes affect the
group’s performance, self-concept, and aspirations and can also drive self-censorship.
Growing up in a segregated, disadvantaged neighborhood can give individuals rules
of thumb that are poorly adapted to success in school and work. The implication is
that equality of outcomes between ascriptive social groups, such as those defined by
race, gender, or ethnicity, should be a policy target along with formal equality of op-
portunity.
Procedural equality of opportunity as a target leaves unaddressed the schema-
tizing power of the institutions that historically denied opportunity to certain
groups. These institutions made the social inequalities appear normal and possibly
normative. A rapidly growing body of literature suggests that interventions can relax
the constraints created by mental models that are a legacy of historical institutions.
The impact of many of the interventions described in this paper are difficult to explain
in rational choice theory, but are not difficult to explain under the quasi-rational, en-
culturated actor framework of behavioral economics.
Notes
Karla Hoff, The World Bank. James Walsh, University of Oxford and The World Bank
1. The first three barriers are obvious, but the fourth may not be. An individual in a low-income class
may rationally choose not to seek upward mobility—for example, not to work hard in school and not to
delay child-bearing beyond adolescence—if doing so would create too great a distance from peers, con-
nections to whom make a person happy (Akerlof, 1997) or if efforts to seek upward mobility (e.g., taking
advanced courses in school or buying a computer) have a high return only if others within one’s social
network make them, too DiMaggio and Garip 2012). Peer group effects create a coordination problem
that can explain the stability of class structure.
2. Bruner (1990).
3. However, some evidence suggests that repeated interactions between ethnic groups in conflict
with each other result in exclusionary attitudes towards the outgroup (Enos 2014).
4. As a result, within an affected school, the presence of poor children increased sharply in new
cohorts but not in existing, older ones. Some schools delayed taking any action on the plan for a year
because enrollment decisions had already been made. The variation— between cohorts within a school
and between schools—in exposure of rich children to poor classmates makes it possible to identify the
impact on the rich children of social interaction with poor children.
5. In a dictator game, there are two players—a dictator and a recipient. The dictator is given an en-
dowment. In this experiment, it was 10 Indian rupees (20 U.S. cents). The dictator makes one decision—
how to split the endowment between himself and the recipient. In this experiment, the students were
invited to play as the dictator in two games. The recipients were anonymous but the dictator had infor-
mation on the socio-economic status of their school. In one game, the recipient was from a school with
poor children. In the other, the recipient was from an elite private school.
6. Boisjoly, Duncan, Kremer, Levy, and Eccles (2006) also find that exposure creates pro-social atti-
tudes and behavior by in-groups toward out-groups. Compared to white students who were not ran-
domly assigned African American roommates in college, white students who were assigned African
American roommates were between one-third and one-half of a standard deviation more likely to en-
dorse affirmative action. They reported several years later that they interacted more often and more
comfortably with minorities.
7. This caused a preference reversal in sessions with both the math-based and verbal-based tasks.
When candidates were made available separately, 65 percent of participants chose lower-performing
males and 44 percent selected higher-performing females. When both a male and female were available
for selection, only 3 percent of the participants chose the lower-performing male and 57 percent chose
the higher-performing female.
8. White students also participated in the study. The intervention improved their performance but
by a smaller amount than that of African American students.
9. The value of crime reduction alone is estimated to yield benefit-cost ratios that range from 5-to-1
up to 30-to-1 ( Heller et al. 2017, p. 5).
10. School engagement increased by 0.14 standard deviations in the first year and by 0.19 standard
deviations the second year.
Hoff and Walsh 27

References
Abel, M., R. Burger, E. Carranza, and P. Piraino. 2017. “Bridging the Intention-Behavior Gap?: The Effect
of Plan-Making Prompts on Job Search and Employment.” Policy Research Working Paper No. 8181,
World Bank, Washington, DC.
Acemoglu, D., T. Reed, and J. Robinson. 2014. “Chiefs: Economic Development and Elite Control of Civil
Society in Sierra Leone.” Journal of Political Economy 122 (2): 319–68.
Acharya, A., M. Blackwell, and M. Sen. 2016. “The Political Legacy of American Slavery.” Journal of
Politics 78 (3): 621–41.
Afridi, F., S. Li, and Y. Ren. 2015. “Social Identity and Inequality: The Impact of China’s Hukou System.”
Journal of Public Economics 123: 17–29.
Akerlof , G., 1976. “The Economics of Caste and of the Rat Race and Other Woeful Tales.” Quarterly
Journal of Economics 90(4): 599–617.
Akerlof , G. 1997. “Social Distance and Social Decisions.” Econometrica 65 (5): 1005–27.
Alesina, A., P. Giuliano, and N. Nunn. 2013. “On the Origin of Gender Roles: Women and the Plough.”
Quarterly Journal of Economics 128 (2): 469–530.
Allport, G. 1950. “Prejudice: A Problem in Psychological and Social Causation.” Journal of Social Issues
6 (S4): 4–23.
Allport, G. 1954. The Nature of Prejudice. Reading, MA: Addison Wesley.
Anderson, E. 1999. Code of the Street. New York: Norton.
Anderson, S., P. Francois, and A. Kotwal. 2015. “Clientelism in Indian Villages.” American Economic Re-
view 105 (6): 1780–816.
Aronson, J., C. B. Fried, and C. Good. 2002. “Reducing the Effects of Stereotype Threat on African Amer-
ican College Students by Shaping Theories of Intelligence.” Journal of Experimental Social Psychology
38 (2): 113–25.
Arrow, K. 1973. “The Theory of Discrimination.” In Discrimination in Labor Markets , edited by O. Ashen-
felter and A. Rees, 3–33. Princeton: Princeton University Press.
Babcock, L., M. P. Recalde, L. Vesterlund, and L. Weingart. 2017. “Gender Differences in Accepting
and Receiving Requests for Tasks with Low Promotability.” American Economic Review 107 (3):
714–47.
Banaji, M., R., and A. G. Greenwald.1995. “Implicit Social Cognition: Attitudes, Self-Esteem, and Stereo-
types.” Psychological Review 102 (1): 4.
Bandura, A. 1986. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs,
NJ: Prentice-Hall.
———. 1997. Self-Efficacy: The Exercise of Control. New York: Freeman.
Beaman, L., E. Duflo, R. Pande, and P. Topalova. 2012. “Female Leadership Raises Aspirations and Edu-
cational Attainment for Girls: A Policy Experiment in India.” Science 335 (6068): 582–6.
Beaman, L., R. Chattopadhyay, E. Duflo, R. Pande, and P. Topalova. 2009. “Powerful Women: Does Ex-
posure Reduce Bias?” Quarterly Journal of Economics 124 (4): 1497–540.
Becker, G. S. 1957. The Economics of Discrimination. Chicago: University of Chicago Press.
Bedolla, L. G., and M. Michelson. 2012. Mobilizing Inclusion: Transforming the Electorate through Get-Out-
the Vote Campaigns. New Haven, CT: Yale University Press.
Bernard, T., S. Dercon, K. Orkin, and A. Taffesse. 2014. “The Future in Mind: Aspirations and Forward-
looking Behavior in Rural Ethiopia.” CSAE Working Paper WPS/2014-16, Centre for Economic Pol-
icy Research, London.
Bertrand, M., D. Chugh, and S. Mullainathan. 2005. “Implicit Discrimination.” American Economic Re-
view 95 (2): 94–98.
Betancourt, T. S., R. McBain, E. A. Newnham, A. M. Akinsulure-Smith, R. T. Brennan, J. R. Weisz, and
N. B. Hansen. 2014. “A Behavioral Intervention for War-Affected Youth in Sierra Leone: A Ran-
domized Controlled Trial.” Journal of the American Academy of Child & Adolescent Psychiatry 53 (12):
1288–97.
Bettinger, E., and R. Baker. 2011. “The Effects of Student Coaching in College: An Evaluation of a Ran-
domized Experiment in Student Mentoring.” NBER Working Paper No. 16881, National Bureau of
Economic Research, Cambridge, MA.
Bicchieri, C. 2006. The Grammar of Society: The Nature and Dynamics of Social Norms. Cambridge: Cam-
bridge University Press.
Blattman, C., J. C. Jamison, and M. Sheridan. 2017. “Reducing Crime and Violence: Experimental Evi-
dence from Cognitive Behavioral Therapy in Liberia.” American Economic Review 107 (4): 1165–206.
Boal, A. 1973. The Theatre of the Oppressed. New York: Routledge Press.
Bohnet, I., A. van Geen, and M. Bazerman. 2016. “When Performance Trumps Gender Bias: Joint versus
Separate Evaluation.” Management Science 62 (5): 1225–34.
Boisjoly, J., G. J. Duncan, M. Kremer, D. M. Levy, and J. Eccles. 2006. “Empathy or Antipathy? The Impact
of Diversity.” American Economic Review 96 (5): 1890–905.
Bourdieu, P. 2000. Pascalian Meditations. Stanford: Stanford University Press.
Broockman, D., and J. Kalla. 2016. “Durably Reducing Transphobia: A Field Experiment on Door-to-
Door Canvassing.” Science 352 (6282): 220–4.
Brooks, B., K. Hoff , and P. Pandey. 2017. “Cultural Impediments to Learning to Cooperate: An Experi-
mental Study of High- and Low-Caste Men in Rural India.”
Bruner, J. S., and M. C. Potter. 1964. “Interference in Visual Recognition.” Science 144 (3617): 424–5.
Bruner, J. 1990. Acts of Meaning. Cambridge, MA: Harvard University Press.
Bruner, J. 1991. “The Narrative Construction of Reality.” Critical Inquiry 18 (1): 1–21.
Camerer, C. F. 2005. “Comments on ‘Development Economics through the Lens of Psychology’ by Send-
hil Mullainathan.” In Annual World Bank Conference on Development Economics: Lessons of Experience,
edited by F. Bourguignon and B. Pleskovic, 71–78. Washington, DC: World Bank.
Carpenter, J., and F. Foos. 2017. “Of UFOs and Politics: How Poor Voters Respond to Policy Promises.”
Manuscript, University of Alabama.
Chetty, R., N. Hendren, and L. Katz. 2016. “The Effects of Exposure to Better Neighborhoods on Children:
New Evidence from the Moving to Opportunity Project.” American Economic Review 106 (4): 855–
902.
Claro, S., D. Paunesku, and C. Dweck. 2016. “Growth Mindset Tempers the Effects of Poverty on Aca-
demic Achievement.” Proceedings of the National Academy of Sciences 113 (31): 8664–8.
Coates, T.-N. 2015. Between the World and Me. New York: Spiegel & Grau.
Coffman, K. B. 2014. “Evidence on Self-Stereotyping and the Contribution of Ideas.” Quarterly Journal
of Economics 129 (4): 1625–60.
Cohen, G. L., J. Garcia, N. Apfel, and A. Master. 2006. “Reducing the Racial Achievement Gap: A Social-
Psychological Intervention.” Science 313 (5791): 1307–10.
Cohen, G. L., J. Garcia, V. Purdie-Vaughns, N. Apfel, and P. Brzustoski. 2009. “Recursive Processes
in Self-Affirmation: Intervening to Close the Minority Achievement Gap.” Science 324 (5925):
400–3.
Hoff and Walsh 29

Cohen, G., and D. K. Sherman. 2014. “The Psychology of Change: Self-Affirmation and Social Psycho-
logical Intervention.” Annual Review of Psychology 65 (1): 333–71.
D’Andrade, R. G. 1995. The Development of Cognitive Anthropology. Cambridge: Cambridge University
Press.
Darley, J., and P. Gross. 1983. “A Hypothesis-Confirming Bias in Labeling Effects.” Journal of Personality
and Social Psychology 44 (1): 20–33.
DiMaggio, P. 1997. “Culture and Cognition.” Annual Review of Sociology 23 (1): 263–87.
DiMaggio, P., and F. Garip. 2012. “Network Effects and Social Inequality.” Annual Review of Sociology 38
(1): 93–118.
Douglas, M. 1966. Purity and Danger. London: Routledge & Kegan Paul.
———. 1986. How Institutions Think. Syracuse, NY: Syracuse University Press.
Dovidio, J. F., and S. L. Gaertner 2004. “Aversive Racism.” Advances in Experimental Social Psychology 36:
1–52.
Dovidio, J. F., K. Kawakami, C. Johnson, B. Johnson, and A. Howard. 1997. “On the Nature of
Prejudice: Automatic and Controlled Processes.” Journal of Experimental Social Psychology 33 (5):
510–40.
Duckworth, A. L., T. Kirby, A. Gollwitzer, and G. Oettingen. 2013. “From Fantasy to Action Mental Con-
trasting with Implementation Intentions (MCII) Improves Academic Performance in Children.” Social
Psychological and Personality 4 (6): 745–53.
Duflo, E. 2012. “Tanner Lectures on Human Values and the Design of the Fight Against Poverty.”
Manuscript, MIT: 28–52.
Dweck, C. 2006. Mindset: The New Psychology of Success. New York: Random House.
Enos, R. D. 2014. “Causal Effect of Intergroup Contact on Exclusionary Attitudes.” Proceedings of the
National Academy of Sciences 111 (10): 3699–704.
Gigerenzer, G., and P. M. Todd, and the ABC Research Group. 1999. Simple Heuristics That Make Us Smart.
New York: Oxford University Press.
Glover, D., A. Pallais, and W. Pariente. 2017. “Discrimination as a Self-Fulfilling Prophecy: Evidence from
French Grocery Stores.” Quarterly Journal of Economics 132 (3): 1219–60.
Goffman, E. 1963. Stigma: Notes on A Spoiled Identity. Englewood Cliffs, NJ: Prentice-Hall.
Goldin, C., and C. Rouse. 2000. “Orchestrating Impartiality: The Impact of ‘Blind’ Auditions on Female
Musicians.” American Economic Review 90 (4): 715–41.
Gollwitzer, P. 1999. “Implementation Intentions: Strong Effects of Simple Plans.” American Psychological
Association 54 (7): 493–503.
Greenwald, A. G., and L. H. Krieger. 2006. “Implicit Bias: Scientific Foundations.” California Law Review
94 (4): 945
Guyon, N., and E. Huillery. 2014. “The Aspiration-Poverty Trap: Why Do Students from Low Social
Background Limit Their Ambition? Evidence from France.” LIEPP Working Paper, Department of
Economics, Sciences Po, Paris.
Heller, S. B., A. K. Shah, J. Guryan, J. Ludwig, S. Mullainathan, and H. A. Pollack. 2017. “Thinking, Fast
and Slow? Some Field Experiments to Reduce Crime and Dropout in Chicago.” Quarterly Journal of
Economics 132 (1): 1–54.
Henrich, J., R. Boyd, S. Bowles, C. Camerer, E. Fehr, H. Gintis, and R. McElreath. 2001. “In Search of
Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies.” American Economic Review
91 (2): 73–78.
Hoff , K., J. Jalan, and Sattwick. 2017. “Can We Tell a New Story? The Impact of Theater for Develop-
ment on Education Aspirations and Domestic Violence.” Presented at the International Economic
Association Conference, Mexico, June.
Hoff , K., M. Kshetramade, and E. Fehr. 2011. “Caste and Punishment: The Legacy of Caste Culture in
Norm Enforcement.” Economic Journal 121 (556): F449–F475.
Hoff , K., and P. Pandey. 2006. “Discrimination, Social Identity, and Durable Inequalities.” American Eco-
nomic Review 96 (2): 206–11.
Hoff , K., and P. Pandey. 2014. “Making Up People: The Effect of Identity on Performance in a Modern-
izing Society.” Journal of Development Economics 106: 118–31.
Hoff , K., and J. E. Stiglitz. 2010. “Equilibrium Fictions: A Cognitive Approach to Societal Rigidity.” Amer-
ican Economic Review 100 (2): 141–46.
Hoff , K., and J. E. Stiglitz. 2016. “Striving for Balance in Economics: Towards a Theory of the Social
Determination of Behavior.” Journal of Economic Behavior and Organization 126 (Part B): 25–57.
Hong, Y.-Y., C. Y. Chiu, C. S. Dweck, D. M. S. Lin, and W. W. N. Wan. 1999. “Implicit Theories, Attribu-
tions, and Coping: A Meaning System Approach.” Journal of Personality and Social Psychology 77 (3):
588.
Iyer, L., A. Mani, P. Mishra, and P. Topalova. 2012. “The Power of Political Voice: Women’s Political
Representation and Crime in India.” American Economic Journal: Applied Economics 4 (4): 165–93.
Jensen, R. 2012. “Do Labor Market Opportunities Affect Young Women’s Work and Family Decisions?
Experimental Evidence from India.” Quarterly Journal of Economics 127 (2): 753–92.
Jodhka, S. S. 2002. “Caste and Untouchability in Rural Punjab.” Economic and Political Weekly 37 (19):
1813–23.
Kahan, D., E. Peters, E. C. Dawson, and P. Slovic. 2017. “Motivated Numeracy and Enlightened Self-
Government.” Behavioral Public Policy 1 (1): 54–86.
Kahneman, D. 2011. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.
La Ferrara, E., A. Chong, and S. Duryea. 2012. “Soap Operas and Fertility: Evidence from Brazil.” Amer-
ican Economic Journal: Applied Economics 4 (4): 1–31.
Leibbrandt, A., and J. A. List. 2014. “Do Women Avoid Salary Negotiations? Evidence from A Large-Scale
Natural Field Experiment.” Management Science 61 (9): 2016–24.
Locke, E. A., and G. P. Latham. 1990. A Theory of Goal Setting & Task Performance. Englewood Cliffs, NJ:
Prentice-Hall.
Markus, H., and S. Kitayama. 2010. “Cultures and Selves: A Cycle of Mutual Constitution.” Perspectives
on Psychological Science 5 (4): 420–30.
Mitchell, J. P., B. Nosek, and M. R. Banaji. 2003. “Contextual Variations in Implicit Evaluation.” Journal
of Experimental Psychology: General 132 (3): 455.
Morisano, D., J. B. Hirsh, J. B. Peterson, R. O. Phil, and B. M. Shore. 2010. “Setting, Elaborating, and
Reflecting on Personal Goals Improves Academic Performance.” Journal of Applied Psychology 95 (2):
255–264.
Namnyak, M., N. Tufton, R. Szekely, M. Toal, S. Worboys, and E.L. Sampson. 2008. “‘Stockholm Syn-
drome’: Psychiatric Diagnosis or Urban Myth?” Acta Psychiatrica Scandinavica 117 (1): 4–11.
Nisbett, R. E., and L. Ross. 1991. The Person and the Situation. New York: McGraw Hill.
Nosek, B., M. Banaji, and A. Greenwald. 2002. “Math = Male, Me = Female, Therefore = Me.” Journal
of Personality and Social Psychology 83 (1): 44–59.
Nosek, B., F. L. Smyth, N. Sriram, N. Lindner, T. Devos, A. Ayala, and Y. Bar-Anan et al. 2009. “Na-
tional Differences in Gender-Science Stereotypes Predict National Sex Differences in Science and
Math Achievement.” Proceedings of the National Academy of Sciences 106 (26): 10593–7.
Hoff and Walsh 31

Nussbaum, D. A., and C. S. Dweck. 2008. “Defensiveness Versus Remediation: Self-Theories and Modes
of Self-Esteem Maintenance.” Personality and Social Psychology Bulletin 34 (5): 599–612.
Nussbaum, M. C. 2001. “Symposium on Amartya Sen’s Philosophy: 5 Adaptive Preferences and
Women’s Options.” Economics and Philosophy 17 (01): 67–88.
Nyhan, B., and J. Reifler, 2010. “When Corrections Fail: The Persistence of Political Misperceptions.”
Political Behavior 32 (2): 303–30.
Oettingen, G. 2014. Rethinking Positive Thinking: Inside the New Science of Motivation. New York: Penguin
Random House.
Okamura, J. Y. 1981. “Situational Ethnicity.” Ethnic and Racial Studies 4 (4): 452–65.
Okonofua, J., D. Paunesku, and G. M. Walton. 2016. “Brief Intervention to Encourage Empathic Dis-
cipline Cuts Suspension Rates in Half among Adolescents.” Proceedings of the National Academy of
Sciences 113 (19): 5221–26.
Pager, D., B. Western, and B. Bonikowski. 2009. “Discrimination in a Low-Wage Labor Market: A Field
Experiment.” American Sociological Review 74 (5): 777–99.
Paunesku, D., G. M. Walton, C. Romero, E. N. Smith, D. S. Yeager, and C. S. Dweck. 2015. “Mind-Set
Interventions are a Scalable Treatment for Academic Underachievement.” Psychological Science 26
(6): 784–93.
Pettigrew, T. F., and L. R. Tropp. 2006. “A Meta-Analytic Test of Intergroup Contact Theory.” Journal of
Personality and Social Psychology 90 (5): 751.
Plante, I., M. Theoret, and O. E. Favreau. 2009. “Student Gender Stereotypes: Contrasting the Perceived
Maleness and Femaleness of Mathematics and Language.” Educational Psychology 29 (4): 385–405.
Pope, D., J. Price, and J. Wolfers. 2013. “Awareness Reduces Racial Bias.” NBER Working Paper No.
19765, National Bureau of Economic Research, Cambridge, MA.
Price, J., and J. Wolfers. 2007. “Racial Discrimination among NBA Referees.” NBER Working Paper No.
13206, National Bureau of Economic Research, Cambridge, MA.
Rao, G. 2018. “Familiarity Does Not Breed Contempt: Diversity, Discrimination and Generosity in Delhi
Schools.” Job Market Paper.
Rocque, M., and R. Paternoster. 2011. “Understanding the Antecedents of the ‘School-to-Jail’ Link: The
Relationship between Race and School Discipline.” Journal of Criminal Law and Criminology 10 (2):
633–66.
Rooth, D.-O. 2010. “Automatic Associations and Discrimination in Hiring: Real World Evidence.” Labour
Economics 17 (3): 523–34.
Sampson, R. J. 2012. “Moving and the Neighborhood Glass Ceiling.” Science 337 (6101): 1464–5.
Sarsons, H. 2017. “Recognition for Group Work: Gender Differences in Academia.” American Economic
Review 107 (5): 141–5.
Schippers, M., A. W. A. Scheepers, and J. B. Peterson. 2015. “A Scalable Goal-Setting Intervention Closes
Both the Gender and Ethnic Minority Achievement Gap.” Palgrave Communications 1: 15014–2015.
Shayo, M., and A. Zussman. 2011. “Judicial Ingroup Bias in the Shadow of Terrorism.” Quarterly Journal
of Economics 126 (3): 1447–84.
Shiv, B., and A. Fedorikhin. 1999. “Heart and Mind in Conflict: The Interplay of Affect and Cognition
in Consumer Decision Making.” Journal of Consumer Research 26 (3): 278–92
Small, D. A., M. Gelfand, L. Babcock, and H. Gettman. 2007. “Who Goes to the Bargaining Table? The
Influence of Gender and Framing on the Initiation of Negotiation.” Journal of Personality and Social
Psychology 93 (4): 600–13.
Steele, C., and J. Aronson. 1995. “Stereotype Threat and the Intellectual Test Performance of African
Americans.” Journal of Personality and Social Psychology 69 (5): 797.
Steele, C., and J. Aronson. 1998. “Stereotype Threat and the Test Performance of Academically Suc-
cessful African Americans.” In The Black-White Test Score Gap, edited by C. Jencks and M. Phillips,
Washington, DC: Brookings Institute.
Strauss, C., and N. Quinn. 1997. A Cognitive Theory of Cultural Meaning. New York: Cambridge University
Press.
Swidler, A. 1986. “Culture in Action: Symbols and Strategies.” American Sociological Review 52 (2): 273–
86.
Thaler, R. H. 2016. “Behavioral Economics: Past, Present, and Future.” American Economic Review 106
(7): 1577–600.
Walton, G. M., and G. L. Cohen. 2007. “A Question of Belonging: Race, Social Fit, and Achievement.”
Journal of Personality and Social Psychology 92 (1): 82.
Wilson, T. D., S. Lindsey, and T. Y. Schooler. 2000. “A Model of Dual Attitudes.” Psychological Review 107
(1): 101.
World Bank. 2015. World Development Report 2015: Mind, Society, and Behavior. Washington, DC: World
Bank.
Wilson, W. J. 1987. The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy. Chicago:
University of Chicago Press.
Wu, A. H., 2017. “Gender Stereotyping in Academia: Evidence from Economics Job Market Rumors Fo-
rum.” Available at SSRN: https://ssrn.com/abstract=3051462.
Yeager, D. S., G. M. Walton, S. T. Brady, E. N. Akcinar, D. Paunesku, L. Keane, and D. Kamentz, et.al.
2016. “Teaching a Lay Theory before College Narrows Achievement Gaps at Scale.” Proceedings of
the National Academy of Sciences 113 (24): E3341–8.
Hoff and Walsh 33

Generalization in the Tropics –
Development Policy, Randomized
Controlled Trials, and External Validity
Jörg Peters, Jörg Langbein, and Gareth Roberts
When properly implemented, Randomized Controlled Trials (RCT) achieve a high degree of
internal validity. Yet, if an RCT is to inform policy, it is critical to establish external valid-
ity. This paper systematically reviews all RCTs conducted in developing countries and pub-
lished in leading economic journals between 2009 and 2014 with respect to how they deal
with external validity. Following Duflo, Glennerster, and Kremer (2008), we scrutinize the
following hazards to external validity: Hawthorne effects, general equilibrium effects, spe-
cific sample problems, and special care in treatment provision. Based on a set of objective
indicators, we find that the majority of published RCTs does not discuss these hazards and
many do not provide the necessary information to assess potential problems. The paper calls
for including external validity dimensions in a more systematic reporting on the results of
RCTs. This may create incentives to avoid overgeneralizing findings and help policy makers to
interpret results appropriately. JEL codes: C83, C93
In recent years, intense debate has taken place about the value of Randomized Con-
trolled Trials (RCTs).1 Most notably in development economics, RCTs have assumed a
dominant role. The striking advantage of RCTs is that they overcome self-selection
into treatment and thus their internal validity is indisputably high. This merit is
sometimes contrasted with shortcomings in external validity (Basu 2014; Deaton
and Cartwright 2016). Critics state that establishing external validity is more diffi-
cult for RCTs than for studies based on observational data (Moffit 2004; Roe and Just
2009; and Temple 2010; Dehejia 2015; Muller 2015; Prittchet and Sandefur 2015).
This is particularly true for RCTs in the development context that tend to be imple-
mented at smaller scale and in a specific locality. Scaling an intervention is likely to
change the treatment effects because the scaled program is typically implemented by
Development / THE WORLD BANK. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com
doi: 10.1093/wbro/lkx005 33:34–64
resource-constrained governments, while the original RCT is often implemented by
effective NGOs or the researchers themselves (Ravallion 2012; Bold et al. 2013;
Banerjee et al. 2017; Deaton and Cartwright 2016).
This does not question the enormous contribution that RCTs have made to existing
knowledge about the effectiveness of policy interventions. Rather, it underscores that
“research designs in economics offer no free lunches—no single approach universally
solves problems of general validity without imposing other limitations,” (Roe and Just
2009). Indeed, Rodrik (2009) argues that RCTs require “credibility-enhancing argu-
ments” to support their external validity—just as observational studies have to make
a stronger case for internal validity. Against this background, the present paper ex-
amines how the results published from RCT-based evaluations are reported, whether
external validity-relevant design features are made transparent, and whether poten-
tial limitations to transferability are discussed.
To this end, we conduct a systematic review of policy evaluations based on RCTs
published in top economic journals. We include all RCTs published between 2009 and
2014 in the American Economic Review, the Quarterly Journal of Economics, Economet-
rica, the Economic Journal, the Review of Economic Studies, the Review of Economics and
Statistics, the Journal of Political Economy and the American Economic Journal: Applied
Economics. In total, we identified 54 RCT-based papers that appeared in these journals.
Since there is no uniform definition of external validity and its hazards in the litera-
ture, in a first step we establish a theoretical framework deducing the assumptions re-
quired to transfer findings from an RCT to another policy population. We do this based
on a model from the philosophical literature on the probabilistic theory of causal-
ity provided by Cartwright (2010), and based on a seminal contribution to the eco-
nomics literature, the toolkit for the implementation of RCTs by Duflo, Glennerster,
and Kremer (2008). We identify four hazards to external validity: (a) Hawthorne and
John Henry Effects; (b) general equilibrium effects; (c) specific sample problems; and
(d) problems that occur when the treatment in the RCT is provided with special care
compared to how it would be implemented under real-world conditions.
As a second step, we scrutinized the reviewed papers with regard to how they deal
with the four external validity dimensions and whether required assumptions are dis-
cussed. Along the lines of these hazards we formulated seven questions, then read all
54 papers carefully with an eye toward whether they address these seven questions.
All questions can be objectively answered by “yes” or “no”; no subjective rating is
involved.
External validity is not necessary in some cases. For example, when RCTs are used
for accountability reasons by a donor or a government, the results are only interpreted
within the evaluated population. Yet, as soon as these findings are used to inform
policy elsewhere or at larger scale, external validity becomes a pivotal element. More-
over, test-of-a-theory or proof of concept RCTs that set out to disprove a general
theoretical proposition speak for themselves and do not need to establish external
Peters et al. 35
validity (Deaton and Cartwright 2016). However, in academic research most RCTs
presumably intend to inform policy, and as we will also confirm in the review, the vast
majority of included papers appear to generalize findings from the study population
to a different policy population.2
Indeed, RCT proponents in the development community advocate in favor of RCTs
in order to create “global public goods” that “can offer reliable guidance to inter-
national organizations, governments, donors, and NGOs beyond national borders,”
(Duflo, Glennerster, and Kremer 2008). As early as 2005, during a symposium on
“New directions in development economics: Theory or empirics?” Abhijit Banerjee
acknowledged the requirement to establish external validity for RCTs and, like Rodrik,
called for arguments that establish the external validity of RCTs (Banerjee 2005). In-
deed, Banerjee and Rodrik seem to agree that external validity is never a self-evident
fact in empirical research, and that RCTs in particular should discuss in how far re-
sults are generalizable.
In the remainder of the paper we first present the theoretical framework and estab-
lish the four hazards to external validity. Following that, the methodological approach
and the seven questions are discussed. The results are presented in the next section,
followed by a discussion section. The subsequent section provides an overview on ex-
isting remedies for external validity problems and ways to deal with them in practice.
The final section concludes.
Theoretical Background and Definition of External Validity

Theoretical Framework
Understanding what external validity exactly is and how it might be threatened is

not clearly defined in the literature. What we are interested in here is the degree to
which an internally valid finding obtained in an RCT is relevant for policy makers who
want to implement the same intervention in a different policy population. Cartwright
(2010) defines external validity in a way that is similar to the understanding conveyed
in Duflo, Glennerster, and Kremer (2008): “External validity has to do with whether
the result that is established in the study will be true elsewhere.” Cartwright provides
a model based on the probabilistic theory of causality. Using this model we identify the
assumptions that have to be made when transferring the results from an RCT to what
a policy maker can expect if she scales the intervention under real-world conditions.
Suppose we are interested in whether a policy intervention C affects a certain out-
come E. We can state that C causes E if
P(E|C&Ki ) > P(E|C̄&Ki )
where Ki describes the environment and intervention particularities under which the
observation is made, and C̄ denotes the absence of the intervention. Assume this
causal relationship was observed in population A and we want to transfer it to a sit-
uation in which C is introduced to another population, A’. In this case, Cartwright
points out that those observations, Ki , have to be identical in both populations A and
A’ as soon as they interfere with the treatment effect. More specifically, Cartwright
formulates the following assumptions that are required: (a) A needs to be a represen-
tative sample of A’; (b) C is introduced in A’ as it was in the experiment in A; (c) the
introduction leaves the causal structure in A’ unchanged.
In the following, we use the language that is widely used in the economics liter-
ature and refer to the toolkit for the implementation of RCTs by Duflo, Glennerster,
and Kremer (2008). Similar to the Cartwright framework, Duflo, Glennerster, and
Kremer introduce external validity as the question “[. . .] whether the impact we mea-
sure would carry over to other samples or populations. In other words, whether the
results are generalizable and replicable”. The four hazards to external validity that are
identified by Duflo, Glennerster, and Kremer are Hawthorne and John Henry Effects,
general equilibrium effects, the specific sample problem, and the special care problem.
The following section presents these hazards to external validity in more detail. Under
the assumption that observational studies mostly evaluate policy interventions that
would have been implemented in every case, Hawthorne/John Henry Effects and the
special care problem are much more likely in RCTs, while general equilibrium effects
and the specific sample problem equally occur in RCTs and observational studies.
Potential Hazards to External Validity
In order to guide the introduction to the different hazards of external validity we use
a stylized intervention of a cash transfer given to young adults in an African village.
Suppose the transfer is randomly assigned among young male adults in the village.
The evaluation examines the consumption patterns of the recipients. We observe that
the transfer receivers use the money to buy some food for their families, football shirts,
and air time for their mobile phones. In comparison, those villagers who did not re-
ceive the transfer will not change their consumption patterns. What would this ob-
servation tell us about giving a cash transfer to people in different set-ups? The an-
swer to this question depends on the assumptions identified in Duflo, Glennerster, and
Kremers’ nomenclature.
Hawthorne and John Henry effects might occur if the participants in an RCT know
or notice that they are part of an experiment and are under observation.3 It is obvi-
ous that this could lead to altered behavior in the treatment group (Hawthorne effect)
and/or the control group (John Henry effect).4 In the stylized cash transfer exam-
ple, the recipient of the transfer can be expected to spend the money for other pur-
poses in case he knows that his behavior is under observation. It is also obvious that
such behavioral responses clearly differ between different experimental set-ups. If the
experiment is embedded into a business-as-usual setup, distortions of participants’
Peters et al. 37
behavior are less likely. In contrast, if the randomized intervention interferes notice-
ably with the participants’ daily life (e.g., an NGO appearing in an African village to
randomize a certain training measure among the villagers), participants will proba-
bly behave differently than they would under non-experimental conditions.5
The special care problem refers to the fact that in RCTs, the treatment is provided
differently from what would be done in a non-controlled program. In the stylized cash
transfer example, a lump sum payment that is scaled up would perhaps be provided by
a larger implementing agency with less personal contact. Bold et al. (2013) provide
compelling evidence for the special care effect in an RCT that was scaled up based on
positive effects observed in a smaller RCT conducted by Duflo, Kremer, and Robinson
(2011b). The major difference is that the program examined in Bold et al. was imple-
mented by the national government instead of an NGO, as was the case in the Duflo
et al. study. The positive results observed in Duflo, Kremer, and Robinson (2011b)
could not be replicated in Bold et al. (2013): “Our results suggest that scaling-up an
intervention (typically defined at the school, clinic, or village level) found to work in
a randomized trial run by a specific organization (often an NGO chosen for its orga-
nizational efficiency) requires an understanding of the whole delivery chain. If this
delivery chain involves a government Ministry with limited implementation capacity
or which is subject to considerable political pressures, agents may respond differently
than they would to an NGO-led experiment.”
Vivalt (2017) confirms the higher effectiveness of RCTs implemented by NGOs or
the researchers themselves as compared to RCTs implemented by governments in a
meta-analysis of published RCTs. Further evidence on the special care problem is pro-
vided by Allcott (2015), who shows that electricity providers that implemented RCTs
in cooperation with a large research program to evaluate household energy conser-
vation instruments are systematically different from those electricity providers that
do not participate in this program. This hints at what Allcott refers to as “site selection
bias”, whereby organizations that agree to cooperate with researchers on an RCT can
be expected to be different compared to those that do not, for example because their
staff are more motivated. This difference could translate into higher general effec-
tiveness. Therefore, the effectiveness observed in RCTs is probably higher than it will
be when the evaluated program is scaled to those organizations that did not initially
cooperate with researchers.
The third identified hazard arises from potential general equilibrium effects (GEE).6
Typically, such GEE only become noticeable if the program is scaled to a broader pop-
ulation or extended to a longer term. In the stylized cash transfer example provided
above, GEE occur if not only a small number of people but many villagers receive
the transfer payment. In this scaled version of the intervention, some of the products
that young male villagers want to buy become scarcer, and thus more expensive. This
also illustrates that GEE can affect non-treated villagers, as prices increase for them
as well. Moreover, in the longer term if the cash transfer program is implemented
permanently, certain norms and attitudes towards labor supply or educational invest-
ment might change.7
This example indicates that GEE in their entirety are difficult to capture. The sever-
ity of GEE, though, depends on some parameters like the regional coverage of the
RCT, the time horizon of the measurements, and the impact indicators that the study
examines. Very small-scale RCTs or those that measure outcomes after a few months
only are unlikely to portray the change in norms and beliefs that the intervention
might entail. Furthermore, market-based outcomes like wages or employment status
will certainly be affected by adjustments in the general equilibrium if an intervention
is scaled and implemented over many years. As a matter of course, it is beyond the
scope of most studies to comprehensively account for such GEE, and RCTs that cleanly
identify partial equilibrium effects can still be informative for policy. A profound dis-
cussion of GEE-relevant features is nonetheless necessary to avoid the ill-advised in-
terpretation of results. Note that GEE are not particular to RCTs and, all else being
equal, the generalizability of the results from observational studies is also exposed by
potential GEE. Many RCTs, particularly in developing country contexts, are however,
limited to a specific region, a relatively small sample size, and short monitoring hori-
zon, and are thus more prone to GEE than country-wide representative panel-data
based observational studies.
In a similar vein, the fourth hazard to external validity, the specific sample problem,
is not particular to RCTs but might be more pronounced in this setting. The problem
occurs if the study population is different from the policy population in which the
intervention will be brought to scale. Taking the cash transfer example, the treatment
effect for young male adults can be expected to be different if the cash transfer is given
to young female adults in the same village or to young male adults in a different part
of the country.
Methods and Data

Review Approach
We reviewed all RCTs conducted in developing countries and published between 2009
and 2014 in the leading journals in economics. We included the five most important
economics journals, namely the American Economic Review, Econometrica, Quarterly
Journal of Economics, Journal of Political Economy, the Review of Economic Studies, as
well as further leading journals that publish empirical work using RCTs such as Amer-
ican Economic Journal: Applied Economics, Economic Journal, and Review of Economics
and Statistics.
We scrutinized all issues in the period, particularly all papers that mention either
the terms “field experiment”, “randomized controlled trials”, or “experimental evi-
dence” in either the title or the abstract, or which indicated in the abstract or the title
Peters et al. 39
Figure 1. Published RCTs Between 2009 and 2014
Note: A total of 54 studies were included, frequencies appear in bold.
that a policy intervention was randomly introduced. We excluded those papers that
examine interventions in an OECD member country.8 In total, 73 papers were initially
identified. Our focus is on policy evaluation and we therefore excluded mere test-of-
a-theory papers.9 In most cases, the demarcation was very obvious and we subse-
quently excluded 19 papers. In total, we found 54 papers based on an RCT to evaluate
a certain policy intervention in a developing country.10 The distribution across jour-
nals is uneven, with the vast majority being published in American Economic Journal:
Applied Economics, American Economic Review and Quarterly Journal of Economics (see
figure 1).
Figure 2 depicts the regional coverage of the surveyed RCTs. The high number
of RCTs implemented in Kenya is due to the strong connection that two of the
most prominent organizations that conduct RCTs have to the country (Innovation
Figure 2. Countries of Implementation
Note: A total of 54 studies were included, frequencies appear in bold.
for Poverty Action [IPA] and the Abdul Latif Jameel Poverty Action Lab [J-Pal]).
Most of these studies were implemented in Kenya’s Western Province by the Dutch
NGO International Child Support (ICS), IPA, and J-Pal’s cooperation partner in the
country.11
We read all 54 papers carefully (including the online supplementary appendix) to
determine whether each paper addressed seven objective yes/no-questions. An ad-
ditional filter question addresses whether the paper has the ambition to generalize.
This is necessary, because it is sometimes argued that not all RCTs intend to generate
generalizable results and are rather designed to test a theoretical concept. In fact, 96
percent of included papers do generalize (see next section for details on the coding
of this question). This is no surprise, since we intentionally excluded test-of-a-theory
papers and focused on policy evaluations. The remaining seven questions all address
the four hazards to external validity outlined in the first, and examine whether the
Peters et al. 41
“credibility-enhancing arguments” (Rodrik 2009) are provided to underpin the plau-
sibility of external validity. Appendix A in the appendix shows the answers to the
seven questions for all surveyed papers individually. In general, we answered the ques-
tions conservatively, that is, when in doubt we answered in favor of the paper. We ab-
stained from applying subjective ratings in order to avoid room for arbitrariness. A
simple report on each paper documents the answers to the seven questions and the
quote from the paper underlying the respective answer. We sent these reports out to
the lead authors of the included papers and asked them to review our answers for
their paper(s).12 For 36 of the 54 papers we received feedback, based on which we
changed an answer from “no” to “yes” in 9 cases (out of 378 questions and answers
in total). The comments we received from the authors are included in the reports, if
necessary followed by a short reply. The revised reports were sent again to the authors
for their information and can be found in the online supplementary appendix to this
paper.
Seven Questions
To elicit the extent the paper accounts for Hawthorne and John Henry effects, we first
asked the following objective questions:
1. Does the paper explicitly say whether participants are aware (or not) of being part
of an experiment or a study?
This question accounts for whether a paper provides the minimum informa-
tion that is required to assess whether Hawthorne and John Henry effects might
occur. More would be desirable: in order to make a substantiated assessment of
Hawthorne-like distortions, information on the implementation of the experi-
ment, the way participants were contacted, which specific explanations they re-
ceived, and the extent to which they were aware of an experiment should be pre-
sented. We assume (and confirmed in the review) that papers that receive a “no”
for question 1 do not discuss these issues because a statement on the participants’
awareness of the study is the obvious point of departure for this discussion. It is
important to note that unlike laboratory or medical experiments, participants in
social science RCTs are not always aware of their participation in an experiment.
Only for those papers that receive a “yes” to question 1 do we additionally pose the
following question:
2. If people are aware of being part of an experiment or a study, does the paper (try
to) account for Hawthorne or John Henry effects (in the design of the study, in the
interpretation of the treatment/mechanisms, or in the interpretation of the size
of the impact)?
The next set of questions probes into general equilibrium effects. As outlined in the
first section, we define general equilibrium effects as changes due to an interven-
tion that occur in a noticeable way only if the intervention is scaled or after a
longer time period.
Two questions capture the two transmission channels through which GEE might
materialize:
3. Does the paper explicitly discuss what might happen if the program is scaled up?
4. Does the paper explicitly discuss if and how the treatment effect might change in
the long run?13
For both questions, we give the answer “yes” as soon as the respective issue is men-
tioned in the paper, irrespective of whether we consider the discussion to be com-
prehensive. The third hazard is what Duflo, Glennerster, and Kremer call the spe-
cific sample problem and is addressed by question 5:
5. Does the paper explicitly discuss the policy population (to which the findings are
generalized) or potential restrictions in generalizing results from the study popu-
lation?
We applied this question only to those papers that explicitly generalize beyond the
study population (see the filter question below). As soon as a paper discusses the
study population vis-à-vis the policy population, we answered the question with
“yes”, irrespective of our personal judgment on whether we deem the statement
to be plausible and the discussion to be comprehensive.
The fourth hazard, special care, is accounted for by the last two questions.
6. Does the paper discuss particularities of how the randomized treatment was pro-
vided in demarcation to a (potential) real-world intervention?
As soon as the paper makes a statement on the design of the treatment compared
to the potential real-world treatment, we answered the question with “yes”, again
irrespective of our personal judgment of whether we deem the statement to be
plausible and comprehensive. In addition, to account for the concern that RCTs im-
plemented by NGOs or researchers themselves might be more effective than scaled
programs implemented by, for example, government agencies, we ask:
7. Who is the implementation partner of the RCT?
The specific wording of the additional filter question is “Does the paper gener-
alize beyond the study population?” Our coding of this question certainly leaves
more room for ambiguity than the coding for the previous objective questions. We
therefore answered this additional question by a “yes” as soon as the paper makes
any generalizing statements (most papers do that in the conclusions) that a mere
test-of-a-theory would not make.14 Note that in this question we do not assess the
Peters et al. 43
Table 1. Reporting on External Validity in Published RCTs
Question Answer is yes (in percent)
Hawthorne and John Henry Effect: Does the paper

1. explicitly say whether participants are aware of being part of an 35
experiment or a study?
2. (try) to account for Hawthorne or John Henry effects?* 29
General Equilibrium Effects: Does the paper
3. discuss what happens if program is scaled up? 44
4. discuss changes to treatment effects in the long run? 46
Specific Sample Problems: Does the paper

5. discuss the policy population or potential restrictions to generalizability? ǂ 77
Special Care: Does the paper

6. cover particularities of how the randomized treatment was provided 20
in demarcation to a (potential) real-world intervention discussed?
Note: * indicates that question 3 only applies to those 19 papers that explicitly state that participants are aware of
being part of an experiment. ǂ indicates that question 5 only applies to those 52 papers that explicitly generalize.
degree to which the paper generalizes (which in fact varies considerably), but only
if it generalizes at all.
Results
Table 1 shows the results for the seven questions asked of every paper. As noted above,
96 percent of the reviewed papers generalize their results. This underpins the pro-
posal that these studies should provide “credibility-enhancing arguments”. It is par-
ticularly striking that only 35 percent of the published papers mention whether peo-
ple are aware of being part of an experiment (question 1). This number also reveals
that it is far from common practice in the economics literature to publish either the
protocol of the experiment or the communication with the participants. Some pa-
pers even mention letters that were sent or read to participants but do not include the
content in the main text or the appendix.
Only 46 percent of all papers discuss how effects might change in the longer term
and whether some sort of adjustments might occur (question 4). Here, it is important
to note that around 65 percent of the reviewed papers examine impacts less than two
years after the randomized treatment; on average, impacts are evaluated 17 months
after the treatment (not shown in the table). While this is in most cases probably
inevitable for practical reasons, a discussion of whether treatment effects might
change in the long run, for example, based on qualitative evidence or theoretical con-
siderations, would be desirable. Note that most of the papers that do discuss long-term
Figure 3. Implementation Partners of Published RCTs
Note: “Regional public authority” refers to interventions implemented by regional governmental entities on the
local level. A total of 54 studies were included, frequencies appear in bold.
effects are those that in fact examine such long-term effects. In other words, only a
small minority of papers that only look at very short-term effects provides a discus-
sion of potential changes in the long run.
Likewise, potential changes in treatment effects in case the intervention is scaled
are hardly discussed (question 3, 44 percent of papers); 35 percent of the papers do
not mention GEE related issues at all, that is, received a “no” for questions 3 and 4
(not shown in table 1). The best score is achieved for the specific sample problem:
77 percent of papers discuss the policy population or potential restrictions to gener-
alizability.
As the results for question 6 show, only 20 percent discuss the special care problem.
This finding has to be interpreted in light of the result for question 7 in figure 3: more
than 60 percent of RCTs were implemented by either the researchers themselves or an
NGO. For these cases, a discussion of the special care issue is particularly relevant. The
Peters et al. 45
remaining RCTs were implemented by either a large firm or a governmental body—
which may better resemble a business-as-usual situation.15
Table A1 in the supplementary online appendix provides a further decomposition
of the results presented in table 1 and shows the share of “yes” answers for the re-
spective year of publication. There is some indication of an improvement from 2009
to 2014, but only for certain questions. For example, the share of “yes”-answers in-
creases to over 50 percent for question 1 on people’s awareness of being part in a
study and question 3 on the implications of scaling. For the specific sample dimen-
sion, the share of “yes” answers to question 5 is lower in 2014 than in it is in the
years from 2009 until 2013. For all other questions, we do not observe major differ-
ences. Overall, there is no clear trend towards a systematic and transparent discussion
of external validity issues.
Discussion
In this section we consider some of the comments and arguments that have been put
forward during the genesis of the paper. We would like to emphasize that for the sake
of transparency and rigor, we only used objective questions and abstained from qual-
itative ratings. While we acknowledge that this approach does not do justice to ev-
ery single paper, we argue that the overall pattern we obtain is a fair representation
of how seriously external validity issues are taken in the publication of RCTs. Please
note once again that we answered all questions very conservatively.
To summarize the results, we find that many published RCTs do not provide a com-
prehensive presentation of how the experiment was implemented.16 More than half
of the papers do not even mention whether the participants in the experiment are
aware of being randomized—which is crucial for assessing whether Hawthorne or
John Henry effects could co-determine the outcomes in the RCT. It is true that in some
cases it is obvious that participants were aware of an experiment, but in most cases it
is indeed ambiguous. In addition, even in cases where it is obvious, it is important to
know what exactly participants were told and thus, a discussion of how vulnerable
the evaluated indicators are to Hawthorne-like distortions would be desirable.
Furthermore, our results show that potential general equilibrium effects are only
rarely addressed. This is above all worrisome in the case that outcomes involve price
changes (e.g., labor market outcomes) so that repercussions when the program is
brought to scale are almost certain. Likewise, the special care problem is hardly dis-
cussed, which is particularly concerning in the developing country context, where
many RCTs are implemented by NGOs that are arguably more flexible in terms of
treatment provision than the government.
A number of good practice examples exist where external validity issues are
avoided by the setting or openly addressed, demonstrating that a transparent dis-
cussion of “credibility enhancing arguments” is possible. As for Hawthorne effects,
in Karlan et al. (2014), for example, participants are not aware of the experiment,
which is also clearly stated in the paper. In Bloom et al. (2013), in contrast, partici-
pants are aware, but the authors discuss the possibility of distorting effects intensely.
For general equilibrium effects, Blattman, Fiala, and Martinez (2014) address po-
tential adjustments in the equilibrium, which are quite likely in their cash transfer
randomization. As for the specific sample problem, Tarozzi et al. (2014) openly dis-
cuss that their study might have taken place in a particular population. Good practice
examples for the special care hazard are again Blattman, Fiala, and Martinez (2014),
since their program is implemented by the government and therefore resembles a
scaled intervention. Duflo, Dupas, and Kremer (2011a) reveal potential special care
problems and acknowledge that a scaled program might be less effective.17
We abstain from giving explicit bad practice examples (for obvious reasons), but
indeed some studies are, we believe, negligently silent about certain hazards in spite
of very obvious problems. In a minority of cases, this is even exacerbated by a very
ambitious and broad generalization of the findings.
Some commentators argued that RCTs that test a theory are not necessarily meant
to be generalized. Yet by design we concentrate our review on papers that evaluate a
certain policy and hence the vast majority of papers included in this review do gen-
eralize results. In addition, a mere test-of-a-theory paper should in our views com-
municate this clearly to avoid misleading interpretations by policy makers and the
public.
This is related to the question of whether in fact all papers are supposed to address
all external validity dimensions included in our review. Our answer is yes, at least for
policy evaluations that generalize their findings. One might argue that some of the
reviewed papers are completely immune to a certain external validity hazard, but the
cost of briefly establishing this immunity is negligible.
Potential Remedies
In an ideal world, external validity would be established by replications in many
different populations and using different designs that vary the parameters which
potentially codetermine the results. Systematic reviews can then compile the col-
lective information in order to identify patterns in the effectiveness that eventually
inform policy. This is the mission of organizations like the Campbell Foundation,
the Cochrane Foundation, as well as the International Initiative for Impact Evalu-
ation (3ie), and systematic reviews have indeed been done in a few cases.18 In a
similar vein, Banerjee et al. (2017) propose a procedure “from proof of concept to
scalable policies.” The authors acknowledge that proof of concept studies are often
intentionally conducted under “ideal conditions through finding a context and im-
plementation partner most likely to make the model work”. These authors suggest
an approach of “multiple iterations of experimentation”, in which the context that
Peters et al. 47
co-determines the results is refined. Banerjee et al. (2017) also provide a promising
example in India for such a scaling up process. Yet it is evident that this approach,
as well as systematic reviews, require a massive collective research endeavor that will
take many years and is probably not feasible in all cases.
It is this paper’s stance that in the meantime, individual RCTs with a claim to
broader policy relevance have to establish external validity, reveal limitations, and
discuss implications for transferability openly. To achieve this goal, the first and most
obvious step is to include a systematic reporting in RCT-based publications follow-
ing the CONSORT statement in the medical literature.19 This reference to the CON-
SORT statement as a role-model for economics research has already been postulated
by Miguel et al. (2014) and Eble, Boone, and Elbourne (2017), for example. Some
design features could be retrieved already in the pre-analysis plan, but at the latest
during the peer-review process the checklist should be included and reviewed. Such
a checklist ensures that the reader has all information at hand allowing her to make
an informed judgment on the transferability of the results. Moreover, potential weak-
nesses should be disclosed, thereby automatically entailing a qualitative discussion
to establish or restrict the study’s external validity. In addition, a mandatory check-
list also creates incentives to already take external validity issues into account in the
study’s design phase.
Next to more transparence in the publication of RCTs, a few instruments exist to
deal with external validity hazards—some of which are post hoc, others of which
can be incorporated in the design of the study. For Hawthorne and John Henry ef-
fects, the most obvious solution is not to inform the participants about the random-
ization, which of course hinges upon the study design. Such an approach resembles
what Levitt and List (2009) refer to as a “natural field experiment”. In some set-ups,
people have to be informed, either because randomization is obvious or for ethical rea-
sons. The standard remedy in medical research—assigning a third group to a placebo
treatment—is not possible in most experiments in social sciences. Aldashev, Kirch-
steiger, and Sebald (2017) emphasize that the assignment procedure that is used to
randomly assign participants into treatment and control groups affects the size of
the bias considerably. These authors suggest that a public randomization reduces bias
compared to a non-transparent private randomization.
Accounting for general equilibrium effects comprehensively is impossible in most
cases, since all types of macro-economic adjustments can hardly be captured in a
micro-economic study. In order to evaluate what eventually happens in the general
equilibrium, one would have to resort to computable general equilibrium (CGE) mod-
els. Indeed, there are ambitions to plug the results of RCT-based evaluations into CGE
models, as is done with observational data in Coady and Harris (2004).
The seminal work on GEE so far tests for the existence of at least selected macro-
economic adjustments and spillovers by randomizing not only the treatment within
clusters (e.g., markets), but also the treatment density between clusters. Influential
examples of this approach are Crépon et al. (2013) on the French labor market, and
Muralidharan and Sundararaman (2015) for school vouchers in India. Using the
same approach, Burke et al. (2017) randomizes the density of loan offers across re-
gions to account for GEE. Moreover, randomizing the intervention on a higher re-
gional aggregation allows for examining the full general equilibrium effect at that
level (Banerjee et al. 2017). Muralidharan, Niehaus, and Sukh (2017), for example,
examine a public employment program at a regional level that is “large enough to
capture general equilibrium effects”. Attanasio, Kugler, and Meghir (2011) exploit
the randomized PROGRESA roll-out on the village level to study GEE on child wages.
As for the specific sample problem, there is an emerging body of literature that
provides guidance on extrapolating findings from one region to another. Pearl and
Bareinboim (2014) develop a conceptual framework that enables the researcher to
decide whether transferring results between populations is possible at all. Moreover,
these authors formulate assumptions that, if they hold true, allow for transferring
results from RCT based studies to observational ones (“license to transport”). Gechter
(2016) takes a similar line and develops a methodology that calculates bounds for
transferring treatment effects obtained in an RCT to a non-experimental sample. The
key assumption here is that “the distribution of treated outcomes for a given un-
treated outcome in the context of interest is consistent with the experimental results,”
(see Gechter 2016). Further contributions offer solutions for very specific types of
RCTs. For example, Kowalski (2016) provides a methodology suitable for RCTs using
an encouragement design (i.e., with low compliance rates), while Stuart et al. (2011)
propose a methodology to account for selection into the RCT sample, which is often
the case in effectiveness studies.
The degree to which scholars believe in the generalizability of results also hinges
upon which part of the results chain they focus. One line of thinking concentrates
on the human behavior component in evaluations, also referred to as “mechanism”,
and assumes this to be more generalizable than what is found on the intervention as a
whole (see, e.g., Bates and Glennerster 2017). The other viewpoint puts more empha-
sis on the treatment as a policy intervention. Here, the complexity of interventions
and the context in which they happen are decisive. This camp calls for combining
evidence from rigorous evaluations with case studies (Woolcock 2013) or “reasoned
intuition” (Basu 2014; Basu and Foster 2015) to transfer findings from one setting
to a different policy population.
This complexity feature is very much related to what we have referred to as
special care in the provision of the treatment, which is arguably very heteroge-
neous across different policy environments. There seems to be a growing consen-
sus that this is an important external validity concern (see, e.g., Banerjee et al.
2017), and some scholars have made recommendations on how to account for
this. Both Bates and Glennerster (2017) and Woolcock (2013) provide frameworks
that guide the transferability assessment, and special care is one important feature.
Peters et al. 49
Bates and Glennerster (2017) suggest isolating the mechanism from other
intervention-related features, while Woolcock (2013) argues that in many “develop-
ing countries [. . .] implementation capability is demonstrably low for logistical tasks,
let alone for complex ones.” Hence, the higher the complexity of an intervention, the
more implementation capability becomes a bottleneck, and, to use our wording, the
more special care puts external validity at risk. Woolcock’s position is that for com-
plex interventions—that is, the vast majority of policy interventions—generalizing
is a “decidedly high-uncertainty undertaking”. Woolcock suggests including quali-
tative case studies into these deliberations.
Conclusion
In theory, there seems to be a consensus among empirical researchers that establish-
ing external validity of a policy evaluation is as important as establishing its internal
validity. Against this background, this paper has systematically reviewed published
RCTs to examine whether external validity concerns are addressed. Our findings sug-
gest that external validity is often neglected and does not play the important role that
it is associated with in review papers and the general academic debate.
In a nutshell, our sole claim is that papers should discuss the extent to which the
different hazards to external validity apply. We call for dedicating the same devotion
to establishing external validity as is done when establishing internal validity. This
thinking implies that papers published in top academic journals are not only tar-
geted to the research community, but also to a policy-oriented audience (including
decision-makers and journalists). This audience, in particular, requires all the infor-
mation necessary to make informed judgments on the extent to which the findings are
transferable to other regions and non-experimental business-as-usual settings. More
transparent reporting would also lead to a situation in which more generalizable RCTs
receive more attention than those that were implemented under heavily-controlled
circumstances or in a very specific region only.
It would be desirable if the peer review process at economics journals explicitly
scrutinized design features of RCTs that are relevant for generalization. As a start-
ing point, this does not need to be more than a checklist and short statements to be
included in an electronic appendix. The logic is that if researchers know already at
the beginning of a study that they will need to provide such checklists and discus-
sions, they will have clear incentives to account for external validity issues in the
study design. Otherwise, external validity degenerates to a nice-to-have feature that
researchers account for voluntarily and for intrinsic reasons. These internal incen-
tives will probably work in many cases. But given the trade-offs we all face during the
laborious implementation of studies, it is almost certain that external validity will of-
ten be sacrificed for other features to which the peer-review process currently pays
more attention.
Notes
Jörg Peters is heading the research group “Climate Change in Developing Countries” at RWI, Germany
and is Professor at University of Passau. All correspondence to be sent to: Jörg Peters, RWI, Hohen-
zollernstraße 1–3, 45128 Essen, Germany, e-mail: peters@rwi-essen.de, phone: 49-201-8149-247.
Jörg Langbein is Researcher at RWI, Germany. Gareth Roberts is lecturer at University of the Witwa-
tersrand and researcher at AMERU, Johannesburg, South Africa. The authors thank Maximilian Hup-
pertz and Julian Rose for excellent research assistance. The authors are also grateful for valuable com-
ments and suggestions by the editor Peter Lanjouw, three anonymous referees, Martin Abel, Mark An-
dor, Michael Grimm, Angus Deaton, Heather Lanthorn, Luciane Lenz, Stephan Klasen, Laura Poswell,
and Colin Vance, as well as seminar participants at the University of Göttingen, University of Passau,
University of the Witwatersrand, and Stockholm Institute of Transition Economics. We contacted all
lead authors of the papers included in this review and many of them provided helpful comments on our
manuscript. Langbein and Peters gratefully acknowledge the support of a special grant (Sondertatbe-
stand) from the German Federal Ministry for Economic Affairs and Energy and the Ministry of Innova-
tion, Science, and Research of the State of North Rhine-Westphalia. A supplemental appendix to this
article is available at https://academic.oup.com/WBRO.
1. The title is an obvious reference to an important contribution to this debate, Angus Deaton’s “In-
struments of Development: Randomization in the Tropics and the Search for the Elusive Keys to Eco-
nomic Development”, published as an NBER working paper in 2009 (Deaton 2009). A revised version
was published under a different title in the Journal of Economic Literature (Deaton 2010).
2. Note that our focus is on policy evaluation. In our protocol, we therefore excluded laboratory ex-
periments, framed field experiments, and test-of-a-theory field experiments that are obviously not meant
to evaluate a policy intervention.
3. The Hawthorne effect in some cases cannot be distinguished from survey effects, the Pygmalion
effect, and the observer-expectancy effect (see Bulte et al. 2014). All of these effects, which generally
also might occur in observational studies, can be amplified by the Hawthorne effect and the experi-
mental character of the study. See Aldashev, Kirchsteiger, and Sebald (2017) for a formalization of the
Hawthorne and John Henry effect.
4. The John Henry effect describes the effect that being randomized into the control group can have
on the performance of control group members. John Henry is a legendary black railroad worker, who—
equipped with a traditional sledgehammer—competed with a steam trill in an experimental setting. Be-
ing aware of this exercise, he strived to outperform the steam drill. While he eventually succeeded, he
died from exhaustion (see Saretsky 1972, for a very classic example of a John Henry effect).
5. See Bulte et al. (2014) and Simons et al. (2017) for evidence on strong Hawthorne effects in ex-
periments in Tanzania and Uganda, respectively, and McCambridge, Witton, and Elbourne (2014) for
a systematic review on Hawthorne effects in medical research. Cilliers, Dube, and Siddiqi (2015) pro-
vide evidence for the distorting effects of foreigner presence in framed field experiments in developing
countries. See also Zwane et al. (2011).
6. See Crépon et al. (2013) for an example of such GEE in a randomized labor market program, in
which treated participants benefited at the expense of non-treated participants.
7. Attanasio, Kugler, and Meghir (2011) observe a reduction in labor supply for child labor in the
Mexican PROGRESA conditional cash transfer intervention, which is disbursed conditioned on children
going to school.
8. The present study builds on an earlier paper that also included RCTs conducted in developed coun-
tries, see Peters, Langbein, and Roberts (2016).
9. See appendix B for the list of the excluded papers and the reason for exclusion.
10. A comprehensive list of included papers and their rating is found in Appendix A.
11. See Roetman (2011) for more information on the genesis of RCTs in Kenya and the role of ICS.
12. The filter question on whether the paper generalizes beyond the study population was added
post-hoc, as a response to comments made by some authors.
Peters et al. 51
13. The time period of a study is of course not only an external validity issue. See King and Behrman
(2009) on the relevance of timing for impact evaluations.
14. We coded this question by “yes” in case the paper derives explicit policy recommendations for
other regions or countries, and in case it makes statements like “our results suggest that this policy
works/does not work” or “our results generalize to”.
15. It could of course be argued that NGOs can also be considered as “business-as-usual”, since
many real-world interventions, especially in developing countries, are implemented by NGOs. How-
ever, for most of the 20 RCTs that were implemented by an NGO, the cooperating NGO was a rather
small one and regionally limited in its activities. Thus, bringing the intervention to scale would be
the task of either the government or a larger NGO with potential implications for the efficacy of the
intervention.
16. This finding is in line with Eble, Boone, and Elbourne (2017) who review RCTs published between
2001 and 2011 for how they deal with different sorts of biases (also covering Hawthorne effects).
17. Details on these examples can be found in the review report on the respective paper in the online
supplementary appendix.
18. Examples of systematic reviews are Acharya et al. (2012) on health insurance for the informal
sector, Evans and Popova (2016) on school learning, Evans and Popova (2017) on cash transfers, and
McKenzie and Woodruff (2013) on the impacts of business training interventions. See also the 3ie sys-
tematic review data base available at: www.3ieimpact.org/en/evidence/systematic-reviews/.
19. See Moher et al. (2010) and Schulz, Altman, and Moher (2010).
References
Acharya, A., S. Vellakkal, F. Taylor, E. Masset, A. Satija, M. Burke, and S. Ebrahim. 2012. “The Impact
of Health Insurance Schemes for the Informal Sector in Low- and Middle-Income Countries: A Sys-
tematic Review.” World Bank Research Observer 28 (2): 236–66.
Adhvaryu, A. 2014. “Learning, Misallocation, and Technology Adoption.” Review of Economic Studies
81 (4): 1331–65.
Aker, J. C., C. Ksoll, and T. J. Lybbert. 2012. “Can Mobile Phones Improve Learning? Evidence from a
Field Experiment in Niger.” American Economic Journal: Applied Economics 4 (4): 94–120.
Alatas, V., A. Banerjee, R. Hanna, B. A. Olken, and J. Tobias. 2012. “Targeting the Poor: Evidence from
a Field Experiment in Indonesia.” American Economic Review 102 (4): 1206–40.
Aldashev, G., G. Kirchsteiger, and A. Sebald. 2017. “Assignment Procedure Biases in Randomised Policy
Experiments.” The Economic Journal 127 (602): 873–95.
Allcott, H. 2015. “Site Selection Bias in Program Evaluation.” Quarterly Journal of Economics 130 (3):
1117–65.
Armantier, O., and A. Boly. 2013. “Comparing Corruption in the Laboratory and in the Field in Burkina
Faso and in Canada.” Economic Journal 123 (573): 1168–87.
Ashraf , N. 2009. “Spousal Control and Intra-household Decision Making: An Experimental Study in the
Philippines.” American Economic Review 99 (4): 1245–77.
Ashraf , N., J. Berry, and J. M. Shapiro. 2010. “Can Higher Prices Stimulate Product Use? Evidence from
a Field Experiment in Zambia.” American Economic Review 100 (5): 2383–413.
Ashraf , N., E. Field, and J. Lee. 2014. “Household Bargaining and Excess Fertility: An Experimental Study
in Zambia.” American Economic Review 104 (7): 2210–37.
Attanasio, O., A. Kugler, and C. Meghir. 2011. “Subsidizing Vocational Training for Disadvantaged
Youth in Colombia: Evidence from a Randomized Trial.” American Economic Journal: Applied Economics
3 (3): 188–220.
Attanasio, O., A. Barr, J. C. Cardenas, G. Genicot, and C. Meghir. 2012. “Risk Pooling, Risk Preferences,
and Social Networks.” American Economic Journal: Applied Economics 4 (2): 134–67.
Banerjee, A. V. 2005. “New Development Economics and the Challenge to Theory.” In New Directions in
Development Economics: Theory or Empirics? A Symposium in Economic and Political Weekly, edited by R.
Kanbur, unpublished paper, August 2005.
Banerjee, A., R. Banerji, J. Berry, E. Duflo, H. Kannan, S. Mukherji, M. Shotland, and M. Walton. 2017.
“From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application.” Journal
of Economic Perspectives 31 (4): 73–102.
Baird, S., C. McIntosh, and B. Özler. 2011. “Cash or Condition? Evidence from a Cash Transfer Experi-
ment.” Quarterly Journal of Economics 126 (4): 1709–53.
Barrera-Osorio, F., M. Bertrand, L. L. Linden, and F. Perez-Calle. 2011. “Improving the Design of Condi-
tional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia.” Ameri-
can Economic Journal: Applied Economics 3 (2): 167–95.
Basu, K. 2014. “Randomisation, Causality and the Role of Reasoned Intuition.” Oxford Development Stud-
ies 42 (4): 455–72.
Basu, K., and A. Foster. 2015. “Development Economics and Method: A Quarter Century of ABCDE.”
World Bank Economic Review 29 (suppl_1): S2–S8.
Bates, M. A., and R. Glennerster. 2017. “The Generalizability Puzzle.” Stanford Social Innovation Review,
Summer 2017: 50–54.
Bauer, M., J. Chytilová, and J. Morduch. 2012. “Behavioral Foundations of Microcredit: Experimental
and Survey Evidence from Rural India.” American Economic Review 102 (2): 1118–39.
Beaman, L., R. Chattopadhyay, E. Duflo, R. Pande, and P. Topalova. 2009. “Powerful Women: Does Ex-
posure Reduce Bias?” Quarterly Journal of Economics 124 (4): 1497–540.
Beaman, L., and J. Magruder. 2012. “Who Gets the Job Referral? Evidence from a Social Networks Ex-
periment.” American Economic Review 102 (7): 3574–93.
Bertrand, M., D. Karlan, S. Mullainathan, E. Shafir, and J. Zinman. 2010. “What’s Advertising Con-
tent Worth? Evidence from a Consumer Credit Marketing Field Experiment.” Quarterly Journal of Eco-
nomics 125 (1): 263–306.
Besley, T. J., K. B. Burchardi, and M. Ghatak. 2012. “Incentives and the De Soto Effect.” Quarterly Journal
of Economics 127 (1): 237–82.
Björkman, M., and J. Svensson. 2009. “Power to the People: Evidence from a Randomized Field Ex-
periment on Community-Based Monitoring in Uganda.” Quarterly Journal of Economics 124 (2):
735–59.
Blattman, C., N. Fiala, and S. Martinez. 2014. “Generating Skilled Self-Employment in Developing Coun-
tries: Experimental Evidence from Uganda.” Quarterly Journal of Economics 129 (2): 697–752.
Blimpo, M. P. 2014. “Team Incentives for Education in Developing Countries: A Randomized Field Ex-
periment in Benin.” American Economic Journal: Applied Economics 6 (4): 90–109.
Bloom, N., B. Eifer, A. Mahajan, D. McKenzie, and J. Roberts. 2013. “Does Management Matter? Evidence
from India.” Quarterly Journal of Economics 128 (1): 1–51.
Bold, T., M. Kimenyi, G. Mwabu, A. Ng’ang’a, and J. Sandefur. 2013. “Scaling up what Works: Exper-
imental Evidence on External Validity in Kenyan Education.” Working Paper No. 321, Center for
Global Development, Washington, DC.
Bulte, E., G. Beekman, S. di Falco, J. Hella, and P. Lei. 2014. “Behavioral Responses and the Impact of new
Agricultural Technologies: Evidence from a Double-Blind Field Experiment in Tanzania.” American
Journal of Agricultural Economics 96 (3): 813–30.
Burde, D., and L. L. Linden. 2013. “Bringing Education to Afghan Girls: A Randomized Controlled Trial
of Village-Based Schools.” American Economic Journal: Applied Economics 5 (3): 27–40.
Peters et al. 53
Burke, M., L. F. Bergquist, and E. Miguel. 2017. “Selling Low and Buying High: An Arbitrage Puz-
zle in Kenyan Villages.” Working Paper, UC Berkeley. Accessed January 17, 2018. Available at:
https://web.stanford.edu/∼mburke/papers/MaizeStorage.pdf .
Bursztyn, L., and L. C. Coffman. 2012. “The Schooling Decision: Family Preferences, Intergenerational
Conflict, and Moral Hazard in the Brazilian Favelas.” Journal of Political Economy 120 (3): 359–97.
Cai, H., Y. Chen, and H. Fang. 2009. “Observational Learning: Evidence from a Randomized Natural
Field Experiment.” American Economic Review 99 (3): 864–82.
Cartwright, N. 2010. “What are Randomised Controlled Trials Good For?” Philosophical Studies 147:
59–70.
Casey, K., R. Glennerster, and E. Miguel. 2012. “Reshaping Institutions: Evidence on Aid Impacts Using
a Pre-Analysis Plan.” Quarterly Journal of Economics 127 (4): 1755–812.
Chassang, S., I. Miquel, G. P., and E. Snowberg. 2012. “Selective Trials: A Principal-Agent Approach to
Randomized Controlled Experiments.” American Economic Review 102 (4): 1279–309.
Chinkhumba, J., S. Godlonton, and R. Thornton. 2014. “The Demand for Medical Male Circumcision.”
American Economic Journal: Applied Economics 6 (2): 152–77.
Cilliers, J., O. Dube, and B. Siddiqi. 2015. “The White-Men Effect: How Foreigner Presence Affects Be-
havior in Experiments.” Journal of Economic Behavior and Organization 118: 397–414.
Coady, D. P., and R. L. Harris. 2004. “Evaluating Transfer Programmes within a General Equilibrium
Framework.” Economic Journal 114 (498): 778–99.
Cohen, J., and P. Dupas. 2010. “Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria
Prevention Experiment.” Quarterly Journal of Economics 125 (1): 1–45.
Collier, P., and P. C. Vicente. 2014. “Votes and Violence: Evidence from a Field Experiment in Nigeria.”
Economic Journal 124 (574): F327–55.
Crépon, B., E. Duflo, M. Gurgand, R. Rathelot, and P. Zamora. 2013. “Do Labor Market Policies have
Displacement Effects? Evidence from a Clustered Randomized Experiment.” Quarterly Journal of Eco-
nomics 128 (2): 531–80.
Das, J., S. Dercon, J. Habyarimana, P. Krishnan, K. Muralidharan, and V. Sundararaman. 2013. “School
Inputs, Household Substitution, and Test Scores.” American Economic Journal: Applied Economics 5 (2):
29–57.
de Mel, S., D. McKenzie, and C. Woodruff . 2009a. “Returns to Capital in Microenterprises: Evidence from
a Field Experiment.” Quarterly Journal of Economics 124 (1): 423.
———. 2009b. “Are Women More Credit Constrained? Experimental Evidence on Gender and Microen-
terprise Returns.” American Economic Journal: Applied Economics 1 (3): 1–32.
———. 2013. “The Demand for, and Consequences of, Formalization among Informal Firms in Sri
Lanka.” American Economic Journal: Applied Economics 5 (2): 122–50.
Deaton, A. S. 2009. “Instruments of Development: Randomization in the Tropics, and the Search for
the Elusive Keys to Economic Development.” NBER Working Paper No. 14690, National Bureau of
Economic Research, Cambridge, MA.
——— 2010. “Instruments, Randomization, and Learning about Development.” Journal of Economic
Literature 48 (2): 424–55.
Deaton, A. S., and N. Cartwright. 2016. “Understanding and Misunderstanding Randomized Controlled
Trials.” NBER Working Paper No. 22595, National Bureau of Economic Research, Cambridge, MA.
Deheija, R. 2015. “Experimental and Non-Experimental Methods in Development Economics: A Porous
Dialectic.” Journal of Globalization and Development 6 (1): 47–69.
DiTella, R., and E. Schargrodsky. 2013. “Criminal Recidivism after Prison and Electronic Monitoring.”
Journal of Political Economy 121 (1): 28–73.
Drexler, A., G. Fischer, and A. Schoar. 2014. “Keeping It Simple: Financial Literacy and Rules of
Thumb.” American Economic Journal: Applied Economics 6 (2): 1–31.
Duflo, E., P. Dupas, and M. Kremer. 2011a. “Peer Effects, Teacher Incentives, and the Impact of Tracking:
Evidence from a Randomized Evaluation in Kenya.” American Economic Review 101 (5): 1739–74.
Duflo, E., R. Glennerster, and M. Kremer. 2008. “Using Randomization in Development Economics Re-
search: A Toolkit.” In Handbook of Development Economics, edited by P. Schultz and J. Strauss, 3895–
962, Amsterdam: North Holland.
Duflo, E., M. Greenstone, R. Pande, and N. Ryan. 2013. “Truth-Telling by Third-Party Auditors and the
Response of Polluting Firms: Experimental Evidence from India.” Quarterly Journal of Economics 128
(4): 1499–545.
Duflo, E., R. Hanna, and S. P. Ryan. 2012. “Incentives Work: Getting Teachers to Come to School.” Amer-
ican Economic Review 102 (4): 1241–78.
Duflo, E., M. Kremer, and J. Robinson. 2011b. “Nudging Farmers to Use Fertilizer: Theory and Experi-
mental Evidence from Kenya.” American Economic Review 101 (6): 2350–90.
Dupas, P. 2011. “Do Teenagers Respond to HIV Risk Information? Evidence from a Field Experiment in
Kenya.” American Economic Journal: Applied Economics 3 (1): 1–34.
Dupas, P. 2014. “Short-Run Subsidies and Long-Run Adoption of New Health Products: Evidence from
a Field Experiment.” Econometrica 82 (1): 197–228.
Dupas, P., and J. Robinson. 2013a. “Saving Constraints and Microenterprise Development: Evidence
from a Field Experiment in Kenya.” American Economic Journal: Applied Economics 5 (1): 163–92.
Dupas, P., and J. Robinson. 2013b. “Why Don’t the Poor Save More? Evidence from Health Savings, Ex-
periments.” American Economic Review 103 (4): 1138–71.
Eble, A., P. Boone, and D. Elbourne. 2017. “On Minimizing the Risk of Bias in Randomized Controlled
Trials in Economics.” World Bank Economic Review 31 (3): 687–707.
Evans, D. K., and A. Popova. 2017. “Cash Transfers and Temptation Goods.” Economic Development and
Cultural Change 65 (2): 189–221.
Evans, D. K., and A. Popova. 2016. “What Really Works to Improve Learning in Developing Countries?
An Analysis of Divergent Findings in Systematic Reviews.” World Bank Research Observer 31 (2): 242–
70.
Feigenberg, B., E. Field, and R. Pande. 2013. “The Economic Returns to Social Interaction: Experimental
Evidence from Microfinance.” Review of Economic Studies 80 (4): 1459–83.
Field, E., R. Pande, J. Papp, and N. Rigol. 2013. “Does the Classic Microfinance Model Discourage En-
trepreneurship among the Poor? Experimental Evidence from India.” American Economic Review 103
(6): 2196–226.
Fujiwara, T., and L. Wantchekon. 2013. “Can Informed Public Deliberation Overcome Clientil-
ism? Experimental Evidence from Benin.” American Economic Journal: Applied Economics 5 (4):
241–55.
Gechter, M. 2016. “Generalizing the Results from Social Experiments: Theory and Evidence.” Working
Paper, Departement of Economics, Boston University. Available at: http://www.personal.psu.edu/
mdg5396/Gechter_Generalizing_Social_Experiments.pdf .
Giné, X., J. Goldberg, and D. Yang. 2012. “Credit Market Consequences of Improved Personal Identifica-
tion: Field Experimental Evidence from Malawi.” American Economic Review 102 (6): 2923–54.
Giné, X., D. Karlan, and K. Zinman. 2010. “Put Your Money Where Your Butt Is: A Commitment Con-
tract For Smoking Cessation.” American Economic Journal: Applied Economics 2 (4): 213–35.
Glewwe, P., N. Ilias, and M. Kremer. 2010. “Teacher Incentives.” American Economic Journal: Applied Eco-
nomics 2 (3): 205–27.
Peters et al. 55
Glewwe, P., M. Kremer, and S. Moulin. 2009. “Many Children Left Behind? Textbooks and Test Scores in
Gneezy, U., K. L. Leonard, and J. A. List. 2009. “Gender Differences in Competition: Evidence from a
Matrilineal and a Patriarchal Society.” Econometrica 77 (5):1637–64.
Hanna, R., S. Mullainathan, and J. Schwartzstein 2014. “Learning through Noticing: Theory and Evi-
dence from a Field Experiment.” Quarterly Journal of Economics 129 (3): 1311–53.
Hjort, J. 2014. “Ethnic Divisions and Production in Firms.” Quarterly Journal of Economics 129 (4): 1899–
946.
Jensen, R. 2010. “The (perceived) Returns for Education and the Demand for Schooling.” Quarterly Jour-
nal of Economics 125 (2): 515–48.
Jensen, R. 2012. “Do Labor Market Opportunities Affect Young Women’s Work and Family Decisions?
Experimental Evidence from India.” Quarterly Journal of Economics 127 (2): 753–92.
Jensen, R. T., and N. H. Miller. 2011. “Do Consumer Prices Subsidies really Improve Nutrition?” Review
of Economics and Statistics 93 (4): 1205–23.
Karlan, D., R. Osei, I. Osei-Akoto, and C. Udry. 2014. “Agricultural Decisions after Relaxing Credit and
Risk Constraints.” Quarterly Journal of Economics 129 (2): 597–652.
Karlan, D., and M. Valdivia. 2011. “Teaching Entrepeneurship: Impact of Business Training on Micro-
finance Clients and Institutions.” Review of Economics and Statistics 93 (2): 510–27.
Karlan, D., and J. Zinman. 2009. “Observing Unobservables: Identifying Information Asymmetries With
a Consumer Credit Field Experiment.” Econometrica 77 (6): 1993–2008.
King, E. M., and J. R. Behrman. 2009. “Timing and Duration of Exposure in Evaluations of Social Pro-
grams.” World Bank Research Observer 24 (1): 55–84.
Kowalski, A. E. 2016. “How to Examine External Validity within an Experiment.” Mimeo., Available at:
http://www.econ.yale.edu/∼ak669/jep.latest.draft.
Kremer, M., J. Leino, E. Miguel, and A. Peterson Zwane. 2011. “Spring Cleaning: Rural Water Impacts,
Valuation, and Property Rights Institutions.” Quarterly Journal of Economics 126 (1): 145–205.
Kremer, M., E. Miguel, and R. Thornton. 2009. “Incentives to Learn.” Review of Economics and Statistics
91 (3): 437–56.
Levitt, S. D., and J. A. List. 2009. “Field Experiments in Economics: The Past, the Present, and the Future.”
European Economic Review 53 (1): 1–18.
Lucas, A. M., and I. M. Mbiti. 2014. “Effects of School Quality on Student Achievement: Discontinuity
Evidence from Kenya.” American Economic Journal: Applied Economics 6 (3): 234–63.
Macours, K., N. Schady, and R. Vakis. 2012. “Cash Transfers, Behavioral Changes, and Cognitive Devel-
opment in Early Childhood: Evidence from a Randomized Experiment.” American Economic Journal:
Applied Economics 4 (2): 247–73.
Macours, K., and R. Vakis. 2014. “Changing Households’ Investment Behaviour through Social Interac-
tions with Local Leaders: Evidence from a Randomised Transfer Programme.” Economic Journal 124
(576): 607–33.
McCambridge, J., J. Witton, and D. R. Elbourne. 2014. “Systematic Review of the Hawthorne Effect: New
Concepts Are Needed to Study Research Participation Effects.” Journal of Clinical Epidemiology 67 (3):
267–77.
McKenzie, D., and C. Woodruff . 2013. “What Are We Learning from Business Training and Entrepeneur-
ship Evaluations around the Developing World?” World Bank Research Observer 29 (1): 48–82.
Miguel, E., C. Camerer, K. Casey, J. Cohen, K. M. Esterling, A. Gerber, and R. Glennerster, et al. 2014.
“Promoting Transparency in Social Science Research.” Science 343 (6166): 30–31.
Moffit, R. 2004. “The Role of Randomized Field Trials in Social Science Research.” American Behavioral
Scientist 47 (5): 506–40.
Moher, D., S. Hopewell, K. F. Schulz, V. Montori, P. C. Gøtzsche, P. J. Devereaux, D. Elbourne, M. Egger, and
D. G. Altman. 2010. “CONSORT 2010 Explanation and Elaboration: Updated Guidelines for reporting
Parallel Group Randomised Trials.” BMJ 340: c869.
Muller, S. M. 2015. “Causal Interaction and External Validity: Obstacles to the Policy Relevance of Ran-
domized Experiments.” World Bank Economic Review 29: S217–25.
Muralidharan, K., P. Niehaus, and S. Sukhtanker. 2017. “General Equilibrium Effects of (Improving)
Public Employment Programs: Experimental Evidence from India.” NBER Working Paper No. 23838,
National Bureau of Economic Research, Cambridge, MA.
Muralidharan, K., and V. Sundararaman. 2015. “The Aggregate Effect of School Choice: Evidence from
a Two-Stage Experiment in India.” Quarterly Journal of Economics 130 (3): 1011–66.
Muralidharan, K., and S. Venkatesh. 2010. “The Impact of Diagnostic Feedback to Teachers on Student
Learning: Experimental Evidence from India.” Economic Journal 120 (546): F187–203.
Muralidharan, K., and S. Venkatesh. 2011. “Teacher Performance Pay: Experimental Evidence from In-
dia.” Journal of Political Economy 119 (1): 39–77.
Olken, B. A., J. Onishi, and S. Wong. 2014. “Should Aid Reward Performance? Evidence from a Field
Experiment on Health and Education in Indonesia.” American Economic Journal: Applied Economics 6
(4): 1–34.
Oster, E., and R. Thornton. 2011. “Menstruation, Sanitary Products, and School Attendance: Evidence
from a Randomized Evaluation.” American Economic Journal: Applied Economics 3 (1): 91–100.
Pearl, J., and E. Bareinboim. 2014. “External Validity: From Do-Calculus to Transportability across Pop-
ulations.” Statistical Science 29 (4): 579–95.
Peters, J., J. Langbein, and G. Roberts. 2016. “Policy Evaluation, Randomized Controlled Trials, and Ex-
ternal Validity–A Systematic Review.” Economics Letters 147: 51–54.
Pradhan, M., D. Suryadarma, A. Beatty, M. Wong, A. Gaduh, A. Alisjahbana, and R. P. Artha. 2014.
“Improving Educational Quality through Enhancing Community Participation: Results from a Ran-
domized Field Experiment in Indonesia.” American Economic Journal: Applied Economics 6 (2): 105–26.
Prittchet, L., and J. Sandefur. 2015. “Learning from Experiments When Context Matters.” American Eco-
nomic Review 105 (5): 471–5.
Ravallion, M. 2012. “Fighting Poverty One Experiment at a Time: A Review of Abhijit Banerjee and
Esther Duflo’s Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty.” Journal of
Economic Literature 50 (1): 103–14.
Robinson, J. 2012. “Limited Insurance within the Household: Evidence from a Field Experiment in
Rodrik, D. 2009. “The new Development Economics: We shall experiment, but how shall we learn?” In
What Works in Development? Thinking Big and Thinking Small, edited by W. Easterly and J. Cohen, 24–
54. Washington, DC: Brookings Institution Press.
Roe, B. E., and D. R. Just. 2009. “Internal and External Validity in Economics Research: Tradeoffs be-
tween Experiments, Field Experiments, Natural Experiments, and Field Data.” American Journal of
Agricultural Economics 91 (5): 1266–71.
Roetman, E. 2011. “A Can of Worms? Implications of Rigorous Impact Evaluations for Development
Agencies.” 3ie Working Paper, Organisation International Impact Initiative (3ie), New Delhi.
Saretsky, G. 1972. “The OEO PC Experiment and the John Henry Effect.” The Phi Delta Kappan 53 (9):
579–81.
Peters et al. 57
Schulz, K. F., D. G. Altman, and D. Moher. 2010. “CONSORT 2010 Statement: Updated Guidelines for
Reporting Parallel Group Randomised Trials.” BMC Medicine 8 (1): 18.
Simons, A. M., T. Beltramo, G. Blalock, and D. I. Levine. 2017. “Using Unobtrusive Sensors to Measure
and Minimize Hawthorne Effects: Evidence from Cookstoves.” Journal of Environmental Economics and
Management 86: 68–80.
Stuart, E. A., S. R. Cole, C. P. Bradshaw, and P. J. Leaf . 2011. “The Use of Propensity Scores to assess the
Generalizability of Results from Randomized Trials.” Journal of the Royal Statistical Society: Series A
(Statistics in Society) 174 (2): 369–86.
Tarozzi, A., A. Mahajan, B. Blackburn, D. Kopf , L. Krishnan, and J. Yoong. 2014. “Micro-loans,
Insecticide-Treated Bednets, and Malaria: Evidence from a Randomized Controlled Trial in Orissa,
India.” American Economic Review 104 (7): 1909–41.
Temple, J. R. W. 2010. “Aid and Conditionality.” In Handbook of Development Economics, vol. 5, edited by
P. Schultz and J. Strauss, 4417–511. Elsevier: North Holland.
Vicente, P. C. 2014. “Is Vote Buying Effective? Evidence from a Field Experiment in West Africa.” Economic
Journal 124 (574): F356–87.
Vivalt, E. 2017. “How Much Can We Generalize from Impact Evaluations?” Mimeo. Australian
National University. Available at: http://evavivalt.com/wp-content/uploads/How-Much-Can-We-
Generalize.pdf .
Voors, M., E. M. E. Nillesen, P. Verwimp, E. H. Bulte, R. Lensink, and D. P. van Soest. 2012. “Violent
Conflict and Behavior: A Field Experiment in Burundi.” American Economic Review 102 (2): 941–64.
Woolcock, M. 2013. “Using Case Studies to Explore the External Validity of ‘Complex’ Development In-
terventions.” Evaluation 19 (3): 229–48.
Zwane, A. P., J. Zinman, E. Van Dusen, W. Pariente, C. Null, E. Miguel, and M. Kremer et al. 2011. “Being
Surveyed can change later Behavior and related Parameter Estimates.” Proceedings of the National
Academy of Sciences 108 (5): 1821–6.
Appendix A: Reviewed Papers and Ratings
Author
Question 1: Additional Question 5: Response:
Peters et al.
Participants Question 3: Question: Policy Question 7: Feedback
aware of Question 2: Scaled-up Question 4: Does the population or Question 6: Implementa- from the
experiment/ Account for program Long-run paper restrictions Special care tion authors
Author study? HJHE? discussed? discussed? generalize? discussed? discussed? partner? received?
Aker et al. No N/A Yes Yes Yes Yes No NGO Yes

(2012)
Alatas et al. Yes Participants No Yes Yes Yes No Government No
(2012) are NOT
aware
Ashraf et al. Yes Yes No Yes Yes Yes No NGO No
(2010)
Ashraf et al. Yes No No Yes Yes Yes No Researcher No
(2014)
Attanasio et al. No N/A Yes No Yes Yes No Government Yes
(2011)
Baird et al. Yes Yes No No Yes Yes No NGO No
(2011)
Barrera-Osorio Yes No No No Yes No No Regional Yes
et al. (2011) Public
Authority
Bertrand et al. No N/A No No Yes Yes No Firm No
(2010)
Björkman and No N/A Yes Yes No N/A Yes NGO Yes
Svennsson
(2009)
Blattman et al. Yes No Yes Yes Yes Yes No Government Yes
(2014)
Blimpo (2014) Yes No Yes No Yes No No Researcher Yes
Bloom et al. Yes Yes No Yes Yes Yes No Researcher Yes
(2013)
59
60
Author
experi- Account for program Long-run paper restrictions Special care tion authors
Author ment/study? HJHE? discussed? discussed? generalize? discussed? discussed? partner? received?
Burde and No N/A No No Yes No No NGO No

Linden (2013)
Casey et al. No N/A No Yes Yes Yes No Government No
(2012)
Chinkhumba et Yes No Yes No Yes Yes No Researcher Yes
al. (2014)
Cohen and No N/A Yes Yes Yes Yes No Researcher Yes
Dupas (2010)
Collier and No N/A No No Yes Yes No NGO Yes
Vicente (2014)
Das et al. (2013) No N/A Yes Yes Yes Yes Yes NGO Yes
de Mel et al. No N/A No No Yes Yes No Researcher No
(2009a)
de Mel et al. Yes No No No Yes Yes No Researcher No
(2013)
Drexler et al. No N/A No No Yes No No Firm No
(2014)
Duflo et al. No N/A Yes No Yes Yes Yes NGO Yes
(2011a)
Duflo et al. No N/A Yes No Yes No No NGO Yes
(2011b)
Duflo et al. Yes No No Yes Yes Yes Yes NGO Yes
(2012)
Duflo et al. No N/A No Yes Yes Yes No Regional Yes
(2013) Public
Authority
The World Bank Research Observer, vol. 33, no. 1 (February 2018)
Author
Peters et al.
Dupas (2011) No N/A Yes Yes Yes Yes Yes NGO Yes
Dupas and Yes No Yes Yes Yes Yes No Firm Yes
Robinson
(2013a)
Dupas and Yes No Yes Yes Yes Yes No Researcher Yes
Robinson
(2013b)
Dupas (2014) Yes No Yes Yes Yes No Yes Researcher Yes
Feigenberg et al. No N/A No Yes Yes Yes No Firm Yes
(2013)
Field et al. No N/A Yes Yes Yes Yes No Firm Yes
(2013)
Fujiwara and No N/A Yes No Yes Yes No Researcher Yes
Wantchekon
(2013)
Giné et al. No N/A Yes Yes Yes Yes No Firm Yes
(2010)
Giné et al. No N/A No Yes Yes Yes No Government Yes
(2012)
Glewwe et al. No N/A No No Yes Yes No NGO Yes
(2009)
Glewwe et al. No N/A No No Yes No No NGO Yes
(2010)
Hanna et al. No N/A No Yes Yes No No Researcher No
(2014)
61
62
Author
Jensen (2010) No N/A No Yes Yes Yes No Researcher Yes

Jensen (2012) No N/A No No Yes Yes No Researcher Yes
Jensen and No N/A Yes No Yes No No Government Yes
Miller (2011)
Karlan et al. Yes Participants Yes No Yes Yes Yes Government No
(2014) are NOT
aware
Karlan and No N/A Yes No No N/A Yes NGO No
Valdivia (2011)
Kremer et al. No N/A No No Yes No No NGO No
(2011)
Kremer et al. Yes Yes Yes Yes Yes Yes Yes NGO Yes
(2009)
Macours et al. No N/A No No Yes Yes No Government No
(2012)
Macours and Yes No No No Yes No No Government No
Vakis (2014)
Muralidharan Yes Yes No No Yes Yes No NGO No
and Venkatesh
(2010)
Muralidharan No N/A Yes Yes Yes Yes Yes NGO No
and Venkatesh
(2011)
Olken et al. No N/A Yes Yes Yes Yes No Government Yes
(2014)
Peters et al.
Author
Oster and No N/A No No Yes Yes No Researcher Yes

Thornton
(2011)
Pradhan et al. No N/A No No Yes No No Firm Yes
(2014)
Robinson Yes No No No Yes Yes No Researcher Yes
(2012)
Tarozzi et al. No N/A Yes No Yes Yes Yes Firm Yes
(2014)
Vicente (2014) No N/A No No Yes Yes No Government Yes
63
Appendix B: Excluded Papers and Reason for Exclusion
Author Reason for exclusion
Adhvaryu (2014) Quasi-experiment
Armantier and Boly (2013) Artefactual experiment
Ashraf (2009) Behavioral Field experiment
Attanasio et al. (2012) Artefactual experiment
Bauer et al. (2012) Behavioral Field experiment
Beaman and Magruder (2012) Artefactual experiment
Beaman et al. (2009) Natural experiment
Besley et al. (2012) Theoretical paper
Bursztyn and Coffman (2012) Natural experiment
Cai et al. (2009) Behavioral field experiment
Chassang et al. (2012) Theoretical paper about RCTs
De Mel et al. (2009b) Reply to a previously published article
DiTella and Schargrodsky (2013) Natural experiment
Gneezy et al. (2009) Artefactual experiment
Hjort (2014) Natural experiment
Karlan and Zinman (2009) Behavioral field experiment
Lucas and Mbiti (2014) Quasi-experiment
Voors et al. (2012) Artefactual experiment
Privatization in Developing Countries:
What Are the Lessons of Recent
Downloaded from https://academic.oup.com/wbro/article-abstract/33/1/65/4951686 by Joint Bank-Fund Library user on 08 August 2019

Experience?
Saul Estrin and Adeline Pelletier
This paper reviews the recent empirical evidence on privatization in developing countries,
with particular emphasis on new areas of research such as the distributional impacts of
privatization. Overall, the literature now reflects a more cautious and nuanced evalua-
tion of privatization. Thus, private ownership alone is no longer argued to automatically
generate economic gains in developing economies; pre-conditions (especially the regulatory
infrastructure) and an appropriate process of privatization are important for attaining a pos-
itive impact. These comprise a list which is often challenging in developing countries: well-
designed and sequenced reforms; the implementation of complementary policies; the creation
of regulatory capacity; attention to poverty and social impacts; and strong public communi-
cation. Even so, the studies do identify the scope for efficiency-enhancing privatization that
also promotes equity in developing countries.
There is a large body of literature about the economic effects of privatization. How-
ever, since it was mainly written in the 1990s, there was typically limited emphasis on
issues which have come to the fore more recently, as well as more recent developments
in the evidence about privatization itself, much of it from developing economies. This
motivated us to write this paper, which summarizes the evidence about the impact of
recent privatizations, not only in terms of firms’ efficiency but also with regard to the
effects on income distribution. In addition, we are particularly attentive to the pro-
cess of privatization in developing countries, notably with respect to the regulatory
apparatus enabling successful privatization experiences.
When governments divested state-owned enterprises in developed economies, es-
pecially in the 1980s and 1990s, their objectives were usually to enhance economic
efficiency by improving firm performance, to decrease government intervention and
doi: 10.1093/wbro/lkx007 33:65–102
increase its revenue, and to introduce competition in monopolized sectors (Vickers
and Yarrow 1988). Much of the earlier evidence about the economic impact of pri-
vatization concerned these topics and was based on data from developed countries
and later, transition countries. These findings have been brought together in two pre-
vious surveys, by Megginson and Netter (2001) and Estrin et al. (2009) respectively.

The former assesses the findings of empirical research on the effects of privatization
up to 2000, mainly from developed and middle-income countries, while the latter
concentrates on transition economies including China, over the 1989 to 2006 pe-
riod.1 However, the experiences from the wave of privatizations that have occurred
in developing countries before and since these studies warrant a new examination of
the impact of privatization in the context of the development process.
The tone of the privatization debate has evolved in recent years in interna-
tional financial institutions as privatization activity has shifted towards developing
economies, and as a consequence of the difficulties of implementation and some pri-
vatization failures in the 1980s and 1990s (Jomo 2008). As a result, more empha-
sis in policy-making is now being placed on creating the preconditions for successful
privatization. Thus, in place of a simple pro-privatization bias characteristic of the
Washington consensus (Boycko, Shleifer, and Vishny 1995), it is now proposed that
governments should first provide a better regulatory and institutional framework, in-
cluding a well-functioning capital market and the protection of consumer and em-
ployee rights. In other words, context matters: ownership reforms should be tailor-
made for the national economic circumstances, with strategies for privatization being
adapted to local conditions. The traditional privatization objective of improving the
efficiency of public enterprises also remains a major goal in developing countries, as
does reducing the subsidies to state-owned enterprises (SOEs).
This article therefore reviews the recent evidence on privatization, with an empha-
sis on developing countries. The first section presents some stylized facts. The next
section examines the effects of privatization in terms of firms’ efficiency and perfor-
mance. In the following section, we go on to examine the distributional impacts of
privatization. Policy recommendations are developed in the final section.
Privatization Trends: Stylized Facts

Privatization Trends Since the Late 1980s
The data on privatization prior to 2008 (with a regional breakdown) is sourced from
the World Bank Privatization database but unfortunately this was discontinued in
2008 and no consolidated data is available after that date. Since we have not been able
to find disaggregated data post-2008, we therefore present world aggregates, based on
the Privatization Barometer database.
The early literature focused on developed economies and Western Europe repre-
sented roughly one-third of global privatization proceeds over the period 1977 to
Figure 1. Value of privatisation transactions in developing countries by region, 1988 to 2008

Source: World Bank, Privatization database. Note: comparable data not available after 2008.
2002 (Roland 2008). Even so, many of these deals only concerned minority stakes
of SOEs (Bortolotti and Milella 2008). There were also spectacular numbers of priva-
tizations during the transition process after 1990 in Central and Eastern Europe, with
proceeds totaling $240 billion to 2008, in addition to widespread free or subsidized
allocation of shares in former SOEs (Estrin et al. 2009). The revenues from privati-
zation have been more limited in Africa, the Middle East and South Asia, with total
proceeds below $50 billion for each (see figure 1).2 However, proceeds are on par with
or above Europe once they are expressed as a percentage of GDP.
For the rest of Asia, the picture is rather different. While South Asia has experi-
enced only a limited number of privatizations (especially India), this was not the case
in East Asia, where total privatization proceeds represented 30% of the world’s total
($230 billion) over the 1988 to 2008 period. China, in particular, stands out. Over a
25-year period, the Chinese government has encouraged innovative forms of indus-
trial ownership, especially at the subnational level, that combine elements of collec-
tive and private property (Brandt and Rawski 2008). New private entry and foreign
direct investment have also been encouraged. As a result, by the end of the 1990s,
the non-state sector accounted for over 60% of GDP and state enterprises’ share in
industrial output had declined from 78% in 1978 to 28% in 1999 (Kikeri and Nellis
2004). The OECD estimated the state-owned share of GDP had further declined to
29.7% by 2006 (Lee 2009).
Finally, in Latin America and especially in Chile, large-scale privatization programs
have been launched, especially in the infrastructure sector, starting in 1974 in Chile
and peaking in the 1990s. Between 1988 and 2008, the total privatization proceeds
in Latin America amounted to $220 billion (28% of total world proceeds).
Estrin and Pelletier 67

Figure 2. Worldwide Privatization Revenues 1988 to 2015 (billions of USD)

Note: 2015 is an estimate as August 30, 2015. Source: Privatization Barometer Available at:
http://www.privatizationbarometer.com/.
One needs to be cautious, however, when interpreting the raw data because of dif-
ferences in the size of economies. The differences between the privatization experi-
ence of Africa, Asia, and Europe become less striking when proceeds are normalized
by GDP, though privatization revenue to GDP is high in Latin America, representing,
on average, 0.5% of GDP over the period.
Privatization Trends Since 2008

The five years to 2015 have been marked by the predominant role of China in global
privatizations, while the EU’s share has been below its long-term average of 45% of
the world’s total proceeds, running at only one-third of worldwide totals, on average.
According to the Privatization Barometer (PB) Report 2013–2014, global privatiza-
tion total proceeds exceeded $1.1 trillion from January 2009 to November 2014, with
$544 billion of divested assets between January 2012 and November 2014.3
In addition, the 20-month period beginning in January 2014 witnessed privatiza-
tions totaling $431.4 billion (PB report 2015). This is far more than any comparable
period since the beginning of the privatization programs in the U.K. in the late
1970s (see figure 2), though as noted below, a significant part of this was driven
by the unwinding of positions taken in banks by governments during the financial
crisis.
China has consistently been one of the top privatizers from 2009 to 2015; it was
the second-largest privatizer in 2009 and the first in 2013, 2014, as well as the
8-month period of January to August 2015. Aggregate privatization deals in China
totaled more than $40 billion in both 2013 and 2014 and a spectacular $133.3 bil-
lion in the first eight months of 2015 through 247 sales. The bulk of these privatiza-
tion revenues came from the public and private placement offering of primary shares
by SOEs (PB report 2015). However, the state’s equity ownership stake was gener-

ally only reduced indirectly, by increasing the total number of shares outstanding
(PB report 2015). In fact, Hsieh and Song (2015) have shown that almost half of the
state-owned firms in 2007 and nearly 60 percent of them in 2012 were legally reg-
istered as private firms. The term used in China for this ownership change is that the
large state-owned firms are “corporatized” rather than privatized. The typical form
this “corporatization” takes is that of a minority share traded in the stock market and
merged into a large state-owned conglomerate, the controlling shareholder (Hsieh
and Song 2015).
The next-leading country in terms of privatization proceeds after China is the
United Kingdom, but it is far behind, with total proceeds of $17.2 billion in 2014
(against $7.8 billion in 2009).
In the EU as a whole, with countries addressing their government deficits post-
2008, privatization proceeds rose to a five-year peak in 2013, to $68.0 billion, and
a nine-year peak of $77.6 billion in 2014, while the annualized value of privatiza-
tions during 2015 (based on the first 8 months) reached $63.3 billion. This repre-
sents more than one-third of the global annual totals in 2014, but is only 20.0% of
worldwide totals in the first 8 months of 2015, and lower than the long-run average
EU share of about 44.6% (PB report 2015). This relative decline of EU privatization
proceeds is also reflected in the fact that China alone generated revenues from priva-
tization almost as great as did the EU countries combined during 2015 ($68.0 billion
versus $77.6 billion for China; PB report 2015).
China and India were the two top emerging countries by total privatization rev-
enues in 2015. The five largest single deals outside the developed world in 2014 were
realized in China, with the recapitalization and primary share offering of CITIC Pa-
cific Ltd, the private placement of BOE Technology Group, the primary-share initial
public offering (IPO) of Dalian Wanda Commercial, and finally the primary-share IPO
of CGN Power and of HK Electrical Investments Ltd.
In the following section, we focus on the privatization experience in Africa and
South Asia. While the privatization programs in Eastern Europe, China, and Latin
America are among the most important in terms of total proceeds, a rich literature
already exists discussing them (see Estrin et al. 2009 on transition economies and
Estache and Trujillo 2008 on Latin America). Moreover, while privatization in Latin
America and Eastern Europe culminated in the 1990s, much privatization in Africa
and South Asia is more recent (Roland 2008).

Privatization Patterns in Africa: A Few Countries Only
Privatization programs in sub-Saharan Africa (SSA) occurred in successive waves,

with some countries privatizing much earlier than others (Bennell 1997). The first
group to start such programs in the late 1970s to early 1980s was composed of fran-

cophone West African countries (e.g., Benin, Guinea, Niger, Senegal, and Togo) but
their progress was limited. The second group, both Anglophone and Francophone
countries (Ghana, Nigeria, Ivory Coast, Mali, Kenya, Malawi, Mozambique, Mada-
gascar, and Uganda), started privatizing in the late 1980s. These programs were of-
ten influenced by pressure from the international financial institutions (Nellis 2008)
though, as noted by Bennell (1997), no significant progress was made anywhere
except Nigeria until the late 1990s. The final group, the “late starters”, did not be-
gin to privatize until the early to mid-1990s. Among this group, Tanzania, Burkina
Faso and Zambia have shown a strong political commitment to privatization, whereas
in the other three countries (Cameroon, Ethiopia, and Sierra Leone), only minimal
progress was made in the 1990s.
Privatization in the 1990s: A Slow Start.

Only a minority of SOEs in SSA were subject to privatization over the period 1991
to 2001, and very little privatization has taken place outside of South Africa,
Ghana, Nigeria, Zambia, and Cote d’Ivoire (Nellis 2008). African states have pri-
vatized a smaller percentage (around 40%) of their SOEs than in Latin America
and the transition economies (Nellis 2008). In addition, privatization has gener-
ally concerned smaller manufacturing, industrial, or service firms. Bennell (1997)
also reports that smaller SOEs were usually targeted during the initial stages
of privatization programs in SSA because they were easier to sell. Five indus-
tries in particular were prominent in most programs: food processing, alcoholic
beverages, textiles, cement and other non-metallic products, and metal products.
These industries accounted for 60% of the total proceeds from the sale of man-
ufacturing SOEs during 1988 to 1995 (Bennell 1997), if we exclude the excep-
tional and large sale of ISCOR (Iron and Steel Industrial Corporation) in South
Africa.
Bennell (1997) explains that the slow progress in privatization in the 1990s was
due to a lack of political commitment compounded by strong opposition from en-
trenched vested interests (senior bureaucrats in ministries and SOEs themselves, as
well as public sector workers concerned about their job security). For instance, in
Cameroon, only five of the thirty SOEs scheduled for privatization were sold by the
end of 1995. In other countries such as Nigeria, the privatization program started
well but then stalled. Despite the fact that Nigeria’s program had been one of the most
successful in SSA in the 1990s, it was suspended in early 1995 in favor of a mass
program of “commercialization”. In Madagascar, the privatization program was also
suspended in mid-1993 due to serious mismanagement and its subsequent unpopu-
larity. In addition, Bennell (1997) reports that there were nationalist concerns about
the possible political and economic consequences of increased foreign ownership as
a result of privatization.
However, in the late 1990s, certain political constraints lifted. First, a growing

number of governments in SSA started to undertake significant economic reforms,
under the aegis of the World Bank and the IMF, in which privatization was an in-
tegral part. Reforms and privatization were also progressively being accepted by the
population. In addition, important political liberalization, with multi-party elections,
broke with the previous statist policies, and created some room for maneuver to im-
plement privatization programs. Finally, the weak financial position of SOEs in many
SSA countries and their rapid deterioration, in conjunction with the fiscal crisis the
state experienced in the 1990s, also opened the way for a sell-off of SOEs to raise gov-
ernment revenues and reduce expenditures.
Despite this stronger commitment, Nellis (2008) notes that there were actually
only a few privatization deals in Africa in the 1990s, mainly in infrastructure, and
even in these the state retained significant minority stakes; around one-third of the
shares on average were retained. Between 1988 and 1999, the total proceeds from
privatization in SSA amounted to $9.8 billion, with the manufacturing and services
sector accounting for 36% of the total, infrastructure 28%, the energy sector 17%,
the primary sector 14%, and the financial (and other) sector 6% (see World Bank Pri-
vatization Database).
The Early to Mid-2000s; More Rapid Progress.

There were some important privatizations in SSA between 2000 and 2008, and to-
tal proceeds increased to $12.654 billion (see World Bank Privatization Database).
Nigeria comprised 51% of this amount, followed by Kenya (10%), Ghana (9%) and
South Africa (6%). Infrastructure4 represented 73% of the total amount of the deals,
followed by the manufacturing and services sector5 (17%), the financial sector6 (6%),
energy7 (4%) and the primary sector8 (1%; see World Bank Privatization Database).
Privatization Post-2008: A Slowdown.

Privatization activity slowed in SSA with the economic downturn after 2008. One
notable exception was Benin, with the privatization of the cotton and the public util-
ity sectors. The concession for the operation of the container terminal of the Port of
Cotonou and the majority stake in the cement company were awarded to a strategic
private investor in September 2009 and March 2010, respectively, and the privatiza-
tion of Benin Telecom was launched in 2009 (this is still ongoing; IMF 2010). Nige-
ria was also notable for its sale of 15 electricity-generating and distribution compa-
nies in 2013, raising $2.50 billion (see Megginson 2014). In Chad, the government

announced in 2015 that it was re-launching the sale of 80% of Société des Telecom-
munications du Tchad (Sotel-Tchad), after the previous attempt collapsed in 2010.
Because the World Bank Privatization Database does not have data on privatization
after 2008, one cannot compare the aggregated privatization proceeds post-2008 to
those of earlier decades.

Privatization in South Asia: A Slow Opening
Privatizations in South Asia have traditionally been rare, despite the notable ineffi-
ciency of SOEs (Gupta 2008). The governments’ reluctance to privatize can be partly
explained historically, with the government’s close involvement in the establishment
of an industrial base in the postcolonial era, especially in India (Gupta 2008). Partic-
ular sectors had been reserved exclusively for SOEs, such as the infrastructure sector
and capital goods and raw materials industries such as steel, petroleum, and heavy
machinery. In addition, the government nationalized many loss-making private com-
panies; more than half of the firms owned by the Indian federal government were
loss-making in the 1990s.
Following the balance of payment crisis of 1991, the Indian government imple-
mented a series of reforms under the Industrial Policy Resolution of 1991 to encour-
age private enterprise. Privatization was initiated mainly through two approaches:
partial privatization and strategic sales. However, the former was very limited, with
the government selling only minority equity stakes until 2000, and without trans-
ferring management control. Political uncertainty prevented the emergence of a
coherent privatization policy. Majority stakes sales and the transfer of management
control were only conducted after the elections of 1999, and even then, until 2004
the government retained an average ownership stake of 82% in all SOEs (Gupta
2008).
The stalled privatization program was revived in 2010 with a secondary offering
of shares in National Thermal Power Corporation Ltd (NTPC), which owns 20% of
India’s power generation capacity (Gupta 2009). However, the sale of the $1.85 bil-
lion block of shares only reduced the government’s stake by an additional 5%, leav-
ing 85% still under government control. In addition, the process of privatization was
viewed as poor, with the secondary offering subscribed only 1.2 times, and even this
after assistance from government-owned financial institutions.
In summary, between 2000 and 2008, the proceeds of privatization in South Asia
totaled $ 17.45 billion, the bulk being realized in India (see figure 3) (55%) followed
by Pakistan (43%). Afghanistan, Bangladesh, Nepal, and Sri Lanka provided the re-
maining 2% (see World Bank Privatization Database). Between 2000 and 2008, the
infrastructure sector represented 51% of the proceeds, followed by the energy sector
(26%), the financial sector (12%), manufacturing and services (10%), and the pri-
mary sector (2%) (see World Bank Privatization Database).
Figure 3. Indian Revenues from Privatization

Source: World Bank Privatization database.
The Effects of Privatization: Efficiency and Firm Performance

Overall, as we report below, the studies on developing economies show that a move
from state to private ownership alone does not automatically yield economic gains.
Rather, a number of factors have been found to influence the success of privatization,
namely:
• Which firms are privatized; there can be a positive (or negative) selection effect.
• Whether privatization is total or partial; evidence suggests that the former is more benefi-
cial.
• The regulatory framework, which in turn depends on the institutional and political envi-
ronment.
• The characteristics of the new owners; foreign ownership has been associated with superior
business performance post-privatization, especially relative to “insider” ownership (priva-
tization to managers and workers).9
• Effective competition. This has been found to be critical in bringing about improvements in
company performance because it is associated with lower costs, lower prices, and higher
operating efficiency.10
In the following sub-sections, we introduce the estimation techniques that have

been used to measure the impact of privatization on firms’ performance, and then ex-
amine privatization experiences in three sectors (banking, telecommunications, and

utilities) in developing countries. We also provide an analysis of the robustness of the
evidence in the literature about the impact of privatization.
Measuring Efficiency and Firms’ Performance Post-Privatization

As Megginson and Sutter (2006) note, researchers face numerous methodological
problems when they analyze the economic effects of privatization. In particular, data
availability and consistency, especially in developing countries, and sample selection
bias—occurring, for example, if the “best” firms are privatized first—represent key
issues. Other problems arise when using accounting data: the determination of the
correct measure of operating performance, the selection of an appropriate bench-
mark and statistical tests are important challenges. These issues are germane to the
interpretation of the results of the studies surveyed below.
A variety of methods have been used to measure the impact of privatization on
firms’ post-privatization performance and efficiency, measured in a number of ways
including return on equity, output growth, labor productivity and changes in cost and
income. We distinguish between two different empirical approaches. The first con-
sists of comparing the performance of government-owned firms to that of privately-
owned firms. The second approach consists of comparing pre-and post-divestment
performance for companies privatized via share issues (public offerings; Megginson,
Nash, and van Randenborgh methodology).
Comparing Government-owned Firms to Privately-owned Firms

An obvious way to examine the impact of privatization is to compare the perfor-
mance of government-owned to privately-owned firms. Studies in this tradition com-
pare post-privatization performance changes with either a comparison group of non-
privatized firms or with a counterfactual. However, important methodological issues
arise, especially in the earlier studies. First, it is difficult to determine the appropriate
set of comparison firms, especially in developing countries where the private sector is
limited. Second, selection effects and endogeneity may bias the comparison, as factors
determining whether the firm is publicly or privately owned are also likely to affect
performance (Gupta, Ham, and Svejnar 2008).
Single Country or Single Industry Comparisons of Costs and Productivity Growth

of Private and Government-Owned Firms.
One of the first studies to compare SOE and private firm performance is that of
Ehrlich et al. (1994). These authors used a sample of 23 comparable interna-
tional airlines (18 from developed countries and 5 from developing/emerging coun-
tries) of different ownership categories over the period 1973 to 1983 for which
they have data on cost and output for comparable goods. These authors find a
significant association between ownership and firm-specific rates of productivity
growth. Interestingly, the empirics also suggest that the benefits derive primarily from
complete privatization of the firm, and that a partial change from state to private
ownership has little effect on long-run productivity growth. Other studies have em-
ployed a similar approach examining differences in efficiency between private and

government-owned firms within a specific country, such as Majumdar (1996) for In-
dian firms and Tian (2000) with Chinese firms. These authors both find that private-
sector firms are more efficient. However, these results are not highly robust from
the perspective of contemporary methods, as they do not directly address selection
issues.
Concerning studies using a counterfactual approach, one can cite the influential
study by Galal et al. (1994), which was sponsored by the World Bank. These authors
compare the actual post-privatization performance of twelve large firms in the air-
lines and utilities industry in Britain, Chile, Malaysia, and Mexico to a counterfac-
tual performance. Further, they estimate net welfare gains in eleven of the twelve
cases considered, equaling on average 26 percent of the firms’ pre-divestiture sales.
La Porta and Lopez-de-Silanes (1999) study privatization in Mexico and find that pri-
vatized Mexican SOEs rapidly close a large performance gap with industry-matched
private firms that had existed prior to divestment. These authors find that output in-
creases by over 50% and that the privatized firms reduce employment by half, while
the remaining workers see a significant pay rise.
Cross-country, Multi-industry Comparisons of X-efficiency and Profitability Ratio

of Private and Government-owned Firms.
Another approach has been to exploit a multi-industry, multi-national cross-
sectional time series to analyze the effects of government ownership on efficiency. The
advantage of this method is that it captures differences that are not apparent in single-
country or single-industry series, and the results are therefore methodologically more
soundly based. In their seminal work, Boardman and Vining (1989) use measures of
X-efficiency and profitability ratios of the 500 largest non-U.S. manufacturing and
mining corporations in 1983 (“The International 500”; Fortune 1983). Privately-
owned firms are found to be significantly more profitable and productive than state-
owned and mixed ownership enterprises, but mixed enterprises are no more profitable
than SOEs. Another important study is that of Frydman et al. (1999), which com-
pares the performance of privatized and state firms in the transition economies of
Central Europe in 1994 using a fixed-effects model. To control for the possibility that
better firms are selected for privatization, these authors compare the pre-privatization
performance of managerially-controlled firms with those controlled by other own-
ers. Frydman et al. (1999) find that privatized firms perform better than the state-
owned firms but that the performance improvement is related to revenue improve-
ment rather than cost reduction in privatized firms.
As noted, governments sequence privatizations strategically, often leading the
most profitable firms to be privatized first (Gupta, Ham, and Svejnar 2008; Dinc and
Gupta 2011). To control for selection and endogeneity biases, the latest studies have
employed more advanced econometric techniques including differences in difference,
triple differences matching methods, and instrumental variable methods.

For instance, Dinc and Gupta (2011) examine the influence of political and finan-
cial factors on the decision to privatize government-owned firms in India using data
from the 1990–2004 period. They find that profitable firms and firms with a lower
wage bill are likely to be privatized early and that the government delays privatiza-
tion in regions where the governing party faces more competition from opposition
parties. The results therefore suggest that firms’ financial characteristics have a sig-
nificant impact on the government’s decision to privatize. This raises an identification
issue for evaluating the effect of privatization on firm performance: if more profitable
firms are more likely to be privatized, we may overstate the impact of privatization on
profitability when we compare the performance of government-owned to that of pri-
vatized firms. The authors then proceed to use political variables as instruments for
the privatization decision, adopting a two-stage least squares treatment effects regres-
sion. After addressing the selection bias, they find that privatization still has a positive
impact on performance in India.
Comparing Pre-post Divestment Sales and Income Data for Companies Privatized
by Public Share Offering
This set of studies examines the effects of privatization on firm performance by

comparing pre- and post-divestment data for companies privatized via public share
offerings. Each firm is compared to itself (a few years earlier) using inflation-adjusted
sales and income data. The first study using this methodology is by Megginson,
Nash, and van Randenborgh (1994). As Megginson and Netter (2001) note, this
methodology suffers from several drawbacks, among which selection bias is probably
the greatest concern, since privatizations through share sales—Share Issue Privati-
zation (SIPs)—represent the largest companies sold during a privatization program.
Another weakness is that the Megginson, Nash, and van Randenborgh methodology
can only examine simple accounting variables (assets, sales, etc.), which is an issue
when comparing accounting information at different points in time and in different
countries. Most of the studies in this tradition also imperfectly account for macroeco-
nomic or industry changes in the pre- and post-privatization window (see Megginson
and Netter 2001, for a critique). These studies also cannot account for the impact on
privatized firms of regulatory or market-opening initiatives that are often launched
in parallel with privatization programs. However, the Megginson, Nash, and van
Randenborgh methodology allows the analysis of large samples of firms from
different industries, countries, and time periods and, while carrying the risk of
selection bias, SIP samples contain the largest and most (politically) important
privatizations.
Most of these studies do identify a significant improvement in company perfor-
mance, post-privatization, though methodological reservations remain. Research in
this tradition has focused on specific industries (banking [Verbrugge, Owens, and

Megginson 2000] and tele-communications [D’Souza and Megginson 2000]); has
used data from a single country (Chile [Maquieira and Zurita 1996]) and employed
multi-industry, multinational samples. However, the significance of many of the op-
erating and financial improvements is not robust to adjustments for changes experi-
enced by other firms over the study period.
A very recent work by Li et al. (2016) overcome the empirical limitations of the
previous SIPs studies mentioned above by employing a triple difference approach.
The authors are able to separate the pure privatization effect from the listing ef-
fect, using a database of 204 Chinese SIPs from 1999 to 2009 matched with other-
wise comparable state-owned enterprises and privately-owned firms. The first double-
difference compares the performance change of SIP firms before and after listing with
the performance change of a control group of fully state-owned and unlisted SOEs to
capture the combined “SIP effect” of going public and privatizing. The second double-
difference compares the performance change of privately-owned firms before and af-
ter their listing with the performance change of a control group of privately-owned
firms that remain unlisted. This captures the “pure listing effect”. These authors ob-
tain the “pure privatization effect” by taking the difference between these two double
differences. Interestingly, they continue to find a positive impact from privatization us-
ing this exacting methodology: they find a significant positive increase in profitability
post-SIP in divested Chinese state-owned companies, even after the negative IPO list-
ing effect is taken into account.
Empirical Evidence to Date in Developing Countries
In this section, we summarize the empirical evidence to date about the effects of pri-
vatization on firms’ performance and efficiency in developing countries, drawing on
the discussion of methodology outlined above. The sectors covered include banking,
telecommunications, and utilities. To examine the reliability of the evidence in draw-
ing policy conclusions, we classify the papers reviewed into four categories depending
on the quality of the sample and the robustness of the methods used.
The Banking Sector
The studies reviewed by Clarke, Cull, and Shirley (2005), which focus on develop-
ing countries and employ the Megginson, Nash, and van Randenborgh methodol-
ogy or a stochastic frontier approach, find that bank performance usually improved

after privatization. For instance, Boubakri et al. (2005), applying the Megginson,
Nash, and van Randenborgh methodology to analyze 81 bank privatizations in 22
low- and middle-income countries, find that some measures of performance improved
after privatization, but that this pattern was not common across countries; environ-
mental factors also played a role. The study by Beck, Cull, and Jerome (2005) in Nige-

ria shows that privatization can improve bank performance, even when the macroe-
conomic and regulatory environment is inhospitable and the government sells the
weakest banks. However, Beck, Cull, and Jerome argue that an adverse macroeco-
nomic and regulatory environment reduces the benefits of privatization.11 Azam, Bi-
ais, and Dia (2004) also show (both theoretically and empirically) the benefits of hav-
ing a strong, independent regulatory agency to ensure that privatized banks play an
efficient role in financial development.
The studies surveyed by Clarke, Cull, and Shirley (2005) also find that bank pri-
vatization has a greater positive effect when it is total rather than partial. This result
has been found in transition countries (Bonin, Hasan, and Wachtel 2005) as well as
in Brazil (Beck, Crivelli, and Summerhill 2005) and Nigeria (Beck, Cull, and Jerome
2005)12 . Furthermore, there is evidence that privatization boosts competition in the
banking sector. For instance, Otchere (2005) examines share-issue privatizations in
nine countries using the Megginson, Nash, and van Randenborgh methodology and
finds that rival banks suffered abnormally negative returns following privatization an-
nouncements, which suggests that shareholders expected more intense competition
and lower returns.
Thus, evidence suggests that performance improves more when the government
fully relinquishes control; when banks are privatized to strategic investors rather
than through share issues; and when bidding is open to all, including foreign banks
(Clarke, Cull, and Shirley 2005; Megginson 2005). A more recent paper by Clarke,
Cull, and Fuchs (2009), which examines the privatization of Uganda Commercial
Bank (UCB) to the South African bank Stanbic, shows that these elements of best
practice also apply when the banking sector is concentrated and under-developed.
The government fully relinquished control to a strategic investor in an open sales pro-
cess that allowed foreign participation, and the authors found that profitability im-
proved post-privatization with no evidence that outreach declined. A similar impact
of privatization to a foreign bank has been found in the case study of the privatization
of Tanzania’s national bank of commerce to the Dutch Rabobank (Cull and Spreng
2011).
The Telecommunications Sector
One of the first telecom studies focused on developing countries, by Wallsten (2001),
used a panel of 30 African and Latin American countries from 1984 to 1997 with a
methodology similar to Megginson, Nash, and van Randenborgh. Overall, the author
finds that competition is significantly associated with increases in per capita access
and decreases in costs. However, privatization alone is associated with few benefits,
and is negatively correlated with connection capacity. In addition, privatization only
improves performance when coupled with effective and independent regulation and
increases in competition.

More recently, Gasmi et al. (2013) have examined the impact of privatization of
the fixed-line telecommunications operator on sector performance, analyzing the out-
comes of privatization reforms in a 1985 to 2007 panel dataset on a selection of 108
countries (including OECD countries, Asia, Africa, Latin America). These authors find
that the impact of privatization on sector outcomes (fixed-line deployment, cellular
deployment, labor efficiency, price of fixed-line) was positive in the OECD countries,
Central America, and the Caribbean, and in resource-scarce coastal Africa and Asia.
However, the impact was negative in South America and in African resource-scarce
landlocked countries, and no significance was identified in resource-rich African
countries.
Gasmi et al. (2013) note that countries with successful privatizations have devel-
oped their infrastructure through the creation of appropriate institutional structures
which have improved the effectiveness of infrastructure policies, and that the cov-
erage of networks increased thanks to the additional capital available with privati-
zation. In contrast, privatization outcomes proved to be poor in South America, in
both resource-scarce landlocked African countries and resource-rich African coun-
tries due to weak contractual design and inadequate enforcement of policies in the in-
frastructure sector, as well as insufficient aggregate demand. In the absence of strong
state capacity, competition appeared to be a more effective instrument to foster per-
formance than privatization.
The extent of infrastructure privatization also diverged across regions. While al-
most all OECD countries have privatized their telecommunications utilities, the rate
of privatization is only around 70% in Latin American, Asian, and African resource-
scarce coastal countries. In African resource-scarce landlocked and resource-rich
countries, the percentage of privatized infrastructure in telecommunications is even
lower, at around 40% and 30%, respectively. Overall, the study by Gasmi et al. (2013)
shows that there were limited privatization effects on network expansion, and that
productive efficiency did not increase in all the regions post-privatization. As such,
the authors conclude that there is no unique model of reform for infrastructure sec-
tors.
The Utilities Sector
Turning to water privatization, Estache and Rossi (2002) estimate a stochastic cost
frontier using 1995 data from a sample of 50 water companies in 29 Asian and
Pacific countries. These authors find that efficiency is not significantly different in

private and public companies. Kirkpatrick, Parker, and Zhang (2006) use a question-
naire survey on water utilities in Africa, covering 13 countries and 14 utilities that
reported private sector involvement, and undertake data envelopment analysis and
stochastic cost frontier techniques. These authors do not find strong evidence of per-
formance differences between state-owned water utilities and water utilities involving

some private capital. The authors consider that this result is related to the technol-
ogy of water provision, the costs of organizing long-term concession agreements, and
regulatory weaknesses. In particular, the authors argue that the nature of the prod-
uct severely restricts the potential for competition and therefore the efficiency gains.13
This means rivalry under privatization must derive from the form of competition for
the market—competition to win the contract or concession agreement. But, as the
authors explain, transaction costs can be high in the process of contracting for water
services provision; for example, the costs of organizing the bidding process, monitor-
ing contract performance, and enforcing contract terms where failures are suspected.
The importance of transparent competition for the market to achieve efficiency gains
and prevent the grabbing of assets by political cronies was also evidenced by more re-
cent research by Tan (2012) in the context of private participation in infrastructure
(PPI) in water in Malaysia. The author shows that the efficiency gains of water pri-
vatization (measured by water loss and unit costs) were inconclusive over the period
2001 to 2008. Despite this, and the subsequent renationalization of water assets, PPI
continues to be promoted—it is being recast in the form of management contracts—
because it provides captive rents. This is also evidenced in the “cherry-picking” of
segments and areas for privatization: private sector participation is concentrated in
the more lucrative water treatment segment and higher income states, leaving the
less profitable segments and (more rural) areas to the public sector.
In terms of privatizing electricity, the study of Zhang, Parker, and Kirkpatrick
(2008) provides an econometric assessment using panel data for 36 developing and
transition countries over the period 1985 to 2003. These authors examine the im-
pact of these reforms on generating capacity, electricity generated, labor productivity
in the generating sector, and capacity utilization. They find that, overall, the gains
in economic performance from privatization and regulations are limited, while in-
troducing competition is more effective to stimulate performance. In particular, they
do not find that privatization leads to improved labor productivity or to higher cap-
ital utilization, or to more generating capacity and higher output, except when it is
coupled with the establishment of an independent regulator. The authors conclude
that when competition is weak, an effective regulatory system is needed to stimulate
performance, while the regulation of state-owned enterprises without privatization
is ineffective.
A more recent study by Balza, Jimenez, and Mercado (2013) examines the rela-
tionship between private sector participation, institutional reform, and performance
of the electricity sector in 18 Latin American countries over the last four decades
(1971 to 2010) This also finds that, regardless of the level of private participa-
tion, well-designed and stable sectoral institutions are essential for improving the
performance of the electricity sector. In particular, privatization is robustly associated
with improvements in quality and efficiency, but not with accessibility to the service.
In contrast, regulatory quality is strongly associated with better performance in terms

of both quality and accessibility.
Summary
To bring together this evidence and evaluate its robustness as a basis for policy, we
classify the papers reviewed in this section into four categories depending on the qual-
ity of the sample and the robustness of the methods used. Category I: single country
data, basic statistics, or econometrics (or small sample). Category II: cross-country
data, basic statistics, or econometrics (or small sample). Category III: single coun-
try data, more advanced econometric techniques. Category IV: cross-country data,
advanced econometric techniques. The findings are reported in table 1 and taken
together, provide qualified evidence that privatization can improve company perfor-
mance, including from studies that use the most advanced econometric methods.
Thus, the evidence from empirical studies of privatization in developing coun-
tries suggests that the performance of banks improved significantly after privatization
in many cases. However, the gains from privatization in the utilities sector (electric-
ity and water) have tended to be limited. Finally, concerning the telecommunications
sector, the impact of privatization on efficiency and coverage varies by region. It has
been shown to be positive in Central America and in resource-scarce coastal Africa
and Asia, but negative in South America and in African resource-landlocked coun-
tries. Thus, the impact appears to be context- as well as sector-specific. The main fac-
tors explaining this variation are regulatory quality (and behind that the quality of
institutions), heterogeneity in effective competition, differences in the detail of con-
tractual design, and in the characteristics of the new owners.
Privatization Process: Distributional Impacts

Thomas Piketty’s recent book (2014), which has highlighted the importance of in-
come distribution in the growth process, also discussed the impact of privatization on
capital accumulation. In principle, privatization need not affect the stock of wealth
in an economy, nor its distribution. State-owned firms are public assets which earn
a return for their owners. Provided the assets to be privatized are valued in such a
way that their price represents the discounted sum of the profits to be earned from
them, then privatization means that the state is replacing an income stream with its
discounted capital value in its asset portfolio. At the same time, the private sector is
purchasing an asset which generates its full value over time from its annual earnings.

82
Table 1. Methodology and Classifications of Empirical Papers
Authors Method Data Results Category
Banks
Azam, Measures of performance: log of bank net Africa (Benin, Burkina, Cote d’Ivoire, Mali, Positive impact of foreign ownership on II
Biais and profits/total loans and log of ratio of bad Niger, Senegal, Togo), 1990 to 1997. Small performance of banks, due to more
Dia loans/total loans. Regress the performance sample (49 observations). risk-seeking strategies by foreign owners.
(2004) of banks on the lagged percentage of lagged
foreign ownership (OLS and GLS
specifications).
Beck, Cull Measures of performance: ROA, ROE, NPL. Nigeria. Unbalanced sample of 69 banks Performance improvements following III
and Megginson, Nash, and van Randenborgh with annual data for the period 1990 privatization, but negative effects of the
Jerome methodology: period of eleven years: three through 2001, with a total of 576 continuing minority government
(2005) years before and eight years after observations. ownership on the performance of many
privatization. Nigerian banks.
Beck, Measures of performance: ROE, ROA, Brazil, unbalanced panel of 207 banks with Privatised banks increased their III
Crivelli, overhead costs/assets quarterly data over the period January performance, but not restructured banks.
and Sum- Megginson, Nash, and van Randenborgh 1995 to September 2003, with a total of
merhill method 4,864 observations.
2005 Examines four options: liquidation,
federalization, privatization and
restructuring
Estrin and Pelletier
Table 1. Continued
Bonin, Measures of performance: cost and profit Transition countries (Bulgaria, the Czech Foreign-owned banks are most efficient, IV
Hasan, efficiency, ROA Four ownership types: Republic, Croatia, Hungary, Poland, and and government-owned banks are least
and foreign greenfield, domestic de novo, Romania); 67 different banks from 1994 to efficient. Voucher privatization does not
Wachtel state-owned, privatised. Stochastic frontier 2002 (451 observations). lead to increased efficiency and
2005 analysis (SFA) to estimate bank efficiency. early-privatised banks are more efficient
than later-privatised banks (and no
evidence of selection effect).
Boubakri Measures of performance: ROE, net interest 81 bank privatizations occurring between Profitability increases post-privatization, IV
et al. margin, credit risk. Examine three 1986 and 1998, in 22 low- and but it depends on the type of owner (higher
(2005) categories of controlling owners: foreign middle-income countries. economic efficiency exhibited by banks
investors, local industrial groups, and the owned by local industrial groups and
government itself. Megginson, Nash, and foreign owners).
van Randenborgh methodology on a panel
of banks. Period of seven years: three years
prior to privatization and three years
post-privatization, including the year of
privatization itself).
83
84
Table 1. Continued
Otchere Measures of performance: CAMEL criteria Analyze 21 privatizations (and 65 rival Statistically significant improvement in IV
(2005) (Capital adequacy, Asset quality, banks) from middle- and low-income operating performance for the privatized
Management efficiency, Earnings ability and countries. banks in the pre- and post-privatization
Labor (employment levels and productivity). period, apart from reduction in loan loss
Stock market data. Megginson, Nash, and provisions ratio. One reason for the lack of
van Randenborgh methodology: 3 years improvement might be the continued
pre-privatization operating performance government ownership of these banks.
data and 5 years post privatization.
Examines pre- and post-privatization
operating performance of the privatised
banks relative to that of the rival banks.
Clarke, Measures of performance: ROA, NPL, total Uganda, 1996 to 2005, 555 observations Improvement in profitability and rate of III
Cull and expenses/total assets. Case study of the (quarterly data). credit growth compared to pre-privatization
Fuchs privatization of Uganda Commercial Bank for UCB.
(2009) to Stanbic (South African bank). Employ
regressions that show the evolution of UCB,
Stanbic, and the post-merger bank in terms
of profitability, portfolio quality, operating
efficiency, and credit growth.
Cull and Measures of performance: ROA, NPL. 42 banks operating in Tanzania between Sale to a foreign strategic investor III
Spreng Examines the privatization of National December 1998 and December 2006. (Rabobank from the Netherlands) resulted
(2011) Bank of Commerce. Test whether the in improved profitability and reductions in
privatization of the two successor banks to non-performing loans, along with an
the original National Bank of Commerce increase in the ratio of loans to total assets.
resulted in improved performance.
Table 1. Continued
Estrin and Pelletier

Telecommunications
Wallsten Measures of performance: mainline 1984 to 1997; 30 African and Latin Privatization combined with an IV
(2001) penetration, payphones, connection American countries. independent regulator is positively
capacity, prices for local calls, labour correlated with telecom performance
efficiency. Megginson, Nash, and van measures. No clear benefits of privatization
Randenborgh, includes fixed effects. alone.
Gasmi, et al. Measures of performance: Mainline 1985 to 2007 panel dataset on a selection Performance of privatization depends on IV
(2013) penetration cellular subscription, mainlines of 108 countries (OECD, Asia, Africa, Latin regional factors related to market
per employee, Monthly subscription to fixed, America). profitability, wealth, and geography.
price of cellular. Empirical analysis of the
impact of privatization of the fixed-line
activity of the traditional
telecommunications operator on
output/efficiency/price. Fixed-effect and
random-effect models, DIF-GMM.
Utilities - water
Estache and Stochastic cost frontier 1995; 50 companies; 29 Asian-Pacific Efficiency is not significantly different in IV
Rossi (2002) countries. private companies than in public ones.
Kirkpatrick, Stochastic cost frontier 2000; Africa; 76 observations, including 10 No strong evidence of differences in the IV
Parker, and private-sector operations. performance of state-owned water utilities
Zhang and water utilities involving some private
(2006) capital in Africa.
85
Table 1. Continued
86
Tan Measures of performance: Nonrevenue 1991 to 2010; Malaysia; 13 Malaysian No evidence of improvement in efficiency I
(2012) water (NRW), unit costs, tariffs, water states. and capital investment after privatization.
production capacity (the amount of water
treated for distribution), length of pipes.
Case study (graphs and statistics). Different
ownerships: public ownership,
corporatized, public–private, private.
Utilities - electricity
Zhang, Measures of performance: net electricity Panel data for 36 developing and Competition seems to be most effective to II
Parker, generation per capita of the population, transitional countries, over the period 1985 increase performance. On their own
and Kirk- installed generation capacity per capita of to 2003. privatization and regulation do not lead to
patrick the population, net electricity generation significant improvement in performance.
(2008) per employee in the industry and electricity
generation to average capacity (capacity
utilization). The privatization variable used
in the study was constructed as the
percentage of generating capacity owned by
private investors. Fixed effects (country and
year) to deal with endogeneity.
Balza, Measures of performance: real end-user 1971 to 2012; 18 Latin American countries Countries with higher private investment II
Jimenez, prices for residential electricity (excluding (panel of countries). Country-level analysis. tend to provide more efficient and
and taxes); percentage of households with better-quality electricity services.
Mercado access to electricity; electricity capacity
(2013) generation; and electricity loss as a
percentage of total electricity production.
Privatization measured as the cumulative
investment in the electricity sector as a
percentage of average gross capital
formation in the period 1984 to 2010.
Hence, privatization does not necessarily entail a net transfer of wealth between the
public and private sectors.
However, the privatization process has not always followed these principles of pub-
lic finance (Estrin et al. 2009). In the extreme, as in the programs in the Czech Re-
public or Russia, significant state assets were transferred to private hands at nominal

or zero prices; in effect, a transfer of wealth from the state to the private sector. More
generally, state assets have frequently been undervalued. This may have been in or-
der to make the assets more attractive to the market, or because the SOEs were loss-
making and the short-term requirement to balance the budget dominated long-term
state asset portfolio criteria. In some cases, ideological arguments have also played a
role; Margaret Thatcher and several of her admirers in transition economies viewed
privatization as a policy mechanism for broadening the private ownership of shares
in companies (Estrin 2002). Whatever the motivation, the undervaluation of state
assets leads to a net redistribution of assets from state to private hands. Piketty ar-
gues that this was an important element in relatively larger growth of private wealth
in Britain than in other Western European countries between 1970 and 2010. Fur-
thermore, it was almost certainly a major factor in what he describes as the “consid-
erable growth of private wealth in Russia and Eastern Europe. . . . which led in some
cases to the spectacularly rapid enrichment of certain individuals (I am thinking of
the Russian oligarchs),” (2014).
As the quotation from Piketty makes clear, the impact on income distribution of
privatization depends on how the ownership of the assets is transferred from state
into private hands; both the pricing and to whom the SOEs are privatized. In the ex-
treme case when assets are transferred by voucher to each citizen equally from the
state to private hands at a zero or nominal price, as in the Czech Republic, there is
a transfer from public to private assets equal to the value of the privatized firms, but
the impact on income distribution will be egalitarian because the process transfers
shares to all citizens equally. In contrast, if assets are freely transferred to a single
wealthy individual, the impact will be to severely worsen the distribution of income.
In practice, state-owned assets that are transferred at below their market value are
often also transferred to individuals who are already wealthy, leading to increasing
inequality.
Political factors may play a significant role in this process, with corrupt elites seiz-
ing state assets for themselves, or using them to reward their cronies or political sup-
porters. Thus, rather than being used to improve efficiency, privatization may be em-
ployed by the ruling group as a mechanism to redistribute wealth and resources.
Acemoglu and Robinson (2012) point to the transfer of state assets into the hands
of the governing elite (often associated with the deliberate continuation of monopoly
power) as a mechanism of extractive political institutions; they cite the telecommuni-
cation privatization in Mexico and the huge amount of wealth accumulated by Carlos
Slim ($47 billion in 2016 dollars) as an example.

But negative distributional effects may also occur for reasons of perceived efficiency
enhancement, for example because the state believes that particular private individ-
uals are those most likely to be able to improve company performance. This implies
a trade-off between efficiency and equity objectives in the privatization process. Eq-
uity is supported by processes which engender dispersed ownership, while it is usually

argued that efficiency is driven by concentrated ownership (Estrin 2002). The empir-
ical evidence highlights this trade-off; improvements in the performance of privatized
firms have been found to depend on subsequent ownership arrangements (Djankov
and Murrell 2002). Notably, privatization to concentrated owners, such as to foreign
firms or to small groups of strategic owners, yields greater improvements in perfor-
mance than privatization to the general population via share offerings, or to man-
agers and workers (Estrin et al. 2009).
Birdsall and Nellis (2003) place the issue of the distributional impact of privati-
zation more formally into an efficiency/equity framework. The effect of privatization
on income distribution between taxpayers and the new owners depends both on the
initial price and on the post-sale stream of value produced. There is no unambigu-
ous prediction about the distributional effects of privatization, which will instead de-
pend on initial conditions, the privatization process and the post-privatization politi-
cal and economic environment. Any assessment of the effects should be dynamic and
highly country-specific, depending on the political and economic context and its his-
tory. However, they argue that there is scope for efficiency-enhancing privatization
which also promotes equity in developing countries.
We review below the distributional impacts of privatizations through their effect
on ownership, employment, prices and their fiscal effects (see table 2 for a summary).
A Review of the Distributional Impacts of Privatizations in the Last Decade
Ownership.
As Megginson (2000) notes, in countries that have privatized through asset sales,
the process has frequently been non-transparent and plagued by insider dealing and
corruption. Thus in Russia, the “loans for shares” programs enabled well-connected
financiers to obtain controlling stakes in the country’s most valuable firms for a price
well below their true value (Megginson 2000). Moreover, the distributional impact of
voucher privatizations has also been disappointing; in Russia and the Czech Repub-
lic, the returns on the vouchers were much lower than anticipated, and very small
in comparison to what a very few well-connected groups of people obtained in the
privatization process (Birdsall and Nellis 2003).
Employment.
Privatization can also affect the distribution of income through its impact on em-
ployment. As public enterprises tend to be overstaffed prior to privatization, private
Table 2. Summary of Distributional Impacts of Privatization (Spillovers)
Distributional
impact Progressive effect Regressive effect
Ownership If the sale is conducted in a transparent If the asset is under-priced and rewards political
way, with a wide distribution of cronyism. If the sale is non-transparent.

vouchers with positive returns.
Employment If newly-privatized firms become more The restructuring and consequent
efficient and dynamic, total employment disproportionate layoff of specific categories of
might recover after the initial worker.
restructuring phase
Prices Privatization can lead to a fall in prices if Prices may increase if they were previously
it is accompanied by increased below cost-recovery level.
competition. In addition, if private
management leads to efficiency gains,
some of the savings can be passed on to
consumers.
Access Access may increase if the privatized If the private owner decreases its engagement
business is expanded through in specific market segments that are beneficial
investments. to the poor. In addition, poorer consumers can
see their access reduced if privatization is
accompanied by the end of illegal water and
electricity connections.
Fiscal If it leads to increased access by the poor Privatization may affect real income (net of
to government services funded by new taxes) if it reduces the tax burden differentially
tax flows. across households. Privatization transfers
control rights to private interests and eliminate
public subsidies, benefiting taxpayers but
reducing consumers’ surplus if costs are
increased.
ownership can lead to restructuring and consequently disproportionate redundan-

cies for specific categories of worker (low-skilled, for instance). The study by Chong
and Lopez-de-Silanes (2002) based on a survey of 308 privatized firms (covering
84 countries) over the period 1982 to 2000 showed that employment was reduced
in 78% post-privatization, likely worsening income distribution (Birdsall and Nellis
2003).
That being said, if the newly-privatized firm becomes more efficient, total employ-
ment might recover after the initial restructuring phase. In addition, government-
owned firms that do not privatize may also have to reduce workforce size. Research
by Gupta (2011) on privatization in India covering the 20-year period of 1989
to 2009 shows that privatization increases employment significantly and is not
associated with a decline in employee compensation.14 Moreover, Gupta argues
that an evaluation of the redistribution of wealth from the government to private

owners must also take account of the cost of subsidies to government-owned firms.
However, the employment costs of privatization will be borne by specific groups of
workers, while the benefits, in terms of reduced subsidies, are distributed across tax-
payers. Hence, privatization may face opposition from organized interests who benefit
from maintaining government ownership.

While Gupta’s (2011) work is a single-country study, it has the merit of using more
advanced econometric methods to control for dynamic selection bias by applying firm
fixed effects and comparing privatized firms to a control group of firms that have also
been selected for privatization but have not yet been sold. In addition, the share of
private ownership is introduced with a lag to reduce the possibility of simultaneity
between privatization and performance.
Prices and Access.

Privatization can also have different impacts on income groups through prices and
access to services. First, privatization can lead to a fall in prices if it is accompanied by
increased competition. In addition, if private management leads to efficiency gains,
some of the savings can be passed on to consumers. However, prices may increase if
they were previously below cost-recovery level. The distributional impact depends on
how the consumption of the firms’ goods and services varies by income levels. Ac-
cess may increase if the privatized business is expanded through investments which
could not be undertaken in public ownership. However, private owners may decrease
their engagement in specific, low-return market segments, which may disproportion-
ately affect the poor. Price increases are common following privatization in network
or infrastructure industries, along with increases in the quality of services. On the
one hand, subsidized services tend to benefit relatively wealthy consumers more than
poorer ones; as such, they may be relatively more impacted than the lower-income
segment by privatization. On the other hand, price increases following the privatiza-
tion of electricity and water will increase the burden of poorer consumers, especially
if it is accompanied by the end of illegal water and electricity connections (Birdsall
and Nellis 2003).
Several studies in Latin America have shown that utility privatization has in fact
led to network expansion and increased access to the service by the population, es-
pecially the rural poor (for Peru, see Torero and Pasco-Font 2001; for Argentina,
see Chisari, Estache, and Romero 1999, Delfino and Casarin 2001, and Ennis and
Pinto 2002; for Bolivia, see Barja and Urquiola 2001; for Mexico, see Lopez-Calva and
Rosellon 2002). This increased network coverage has often been the consequence
of market expansion enabled by private investment capital (see Clarke, Kosec, and
Wallsten (2004)).
When access has increased significantly without a steep rise in prices, privati-
zation has had positive distributional effects (Birdsall and Nellis 2003). However,
increased access has often been accompanied by substantial price increases (Estache,
Foster, and Wodon 2002). In addition, an important negative distributional impact
has been realized through the elimination of illegal connections to electricity and wa-
ter networks by lower-income people. A recent paper by Hailu, Guerreiro-Osorio, and
Tsukada (2012) on water service privatization in Bolivia in the late 1990s and early

2000s shows how tariff increases required for full cost recovery may lead to adverse
privatization outcomes; in this case, the eventual renationalization of the company.
To examine the impact of privatization on access, the authors use a difference-in-
difference approach comparing two groups: households in cities where the utility was
privatized, and households in other cities, with two points in time, before (1996) and
after (2001 and 2005) privatization. These authors find a positive relationship be-
tween access to water and living in cities where the water utility was privatized. How-
ever, the water sector was renationalized in 2006, partly because of popular move-
ments against the tariff increases required for full cost recovery and the failure of the
concessionaire to meet targets stipulated in the contract.
Finally, Austin, Descisciolo, and Samuelsen (2016) point to the limits of privati-
zation in sectors with public goods characteristics. Examining the privatization of
healthcare in 99 less-developed nations over the 1995–2000 period, they employ
two-way fixed effects ordinary least squares regression models. The fixed effects al-
low them to deal with unmeasured, time-invariant variables that are excluded from a
regression model. They regress tuberculosis prevalence per 100,000 on the log of pri-
vate health expenditures, the log of public health expenditures and a set of controls
(economic development, education, HIV prevalence and access to water and sanita-
tion). They find that, while public health expenditures reduce tuberculosis rates in
developing nations over time, this is not the case for private health expenditures.
Fiscal Effects.
The fiscal effects of privatization on income distribution are indirect and come
through changes in revenues and expenditures. In particular, privatization may af-
fect real income (net of taxes) if it reduces the tax burden differentially across house-
holds, or if it leads to increased access by the poor to government services funded
by new tax flows. The study of Davis et al. (2000) on 18 developing and transition
countries showed that the net fiscal effects of privatization were receipts in the order
of 1% of GDP. In some countries, the main fiscal benefits of privatization have been
to eliminate subsidies. Subsidies in critical infrastructure services have often led to
the rationing of under-priced services, hardly affecting poorer households that often
had little or no access to these services, while the non-poor enjoyed the underpriced
access. To the extent that privatization stops these flows of subsidies, it produces
indirect benefits in terms of increased retained revenues (Birdsall and Nellis 2003),
which could indirectly benefit the poor.

Policy Implications
The traditional literature, primarily concerning developed economies, argued that
privatization had largely positive effects on the economic and financial performance
of the companies involved, as well as wider spillover benefits, for example, via techno-

logical diffusion from foreign ownership of former SOEs and enhanced efficiency from
the privatization of utilities and other forms of infrastructure. Moreover, privatization
programs also frequently achieved additional objectives, including the generation
of revenues to relax state budget constraints and a broadening of share ownership
amongst the population. On this basis, privatization became an important element of
reform programs in transition and then developing economies from the 1990s. The
experience of the past twenty years leaves some of these conclusions unchanged, but
leads us to a more nuanced evaluation of the effects of privatization in the context of
economic development.
In particular, though state sectors are often very large in developing economies, it
has been hard to establish widespread privatization programs in many parts of the
world, partly because of political opposition. This has arisen for a variety of reasons.
First, the record of privatization as it spread to middle income and then transition
economies (including China) was not always so positive as in developed economies.
The lesson of the transition economy experience was that privatization was not al-
ways a panacea: if the mode of privatization was inappropriate or the market environ-
ment not competitive, privatization might not enhance the performance of the firms
involved (Estrin et al. 2009). Moreover, privatization programs were associated with
scandals: inappropriate valuations led to the emergence of extreme inequalities of
wealth. Second, in developing economies where the institutional environment, par-
ticularly with respect to regulation of monopolies, was sometimes even weaker than
in transition economies, the benefits of privatization were even less automatic, de-
pending on the sector, and were contingent to a significant degree on the design of
the privatization program. Third, distributional issues are especially significant in de-
veloping economies, so privatization programs also had to consider distributional im-
pacts in ways that had been less relevant for developed economies; opposition rested
on issues raised by the efficiency-equity trade-off. Finally, political economy issues are
perhaps of even greater consequence for policy choices in developing economies, and
privatization programs are especially open to manipulation by extractive political in-
stitutions and elites in fragmented political environments.
This long list of concerns has meant that the spread of privatization programs
to developing countries has been limited, both geographically and with respect to
sectoral reach. The slowdown in privatization has no doubt been exacerbated by
the global recession of 2008 and the resulting flight from risk, which has particu-
larly affected stock markets in developing economies. Moreover, the evidence about
the effects of such privatizations of economic performance is quite nuanced. To be
successful, a privatization program needs to align its objectives with its methods of
privatization, taking into account the sector in which the company operates and the
national, institutional, and political context.

Necessary Pre-conditions for Successful Interventions: Regulatory Agencies and
Managerial Incentives
As Lopez-de-Silanes (2005) notes, good rules and contracts are key for a smooth and
beneficial privatization process. However, government restructuring of SOEs prior to
their sale is likely to be fraught with political difficulties because officials may try to ex-
tract private benefits. Although restructuring could increase revenues from the sale,
Lopez-de-Silanes suggests that restructuring policies do not lead to higher revenues.
In addition, Lopez-de-Silanes (2005) notes the importance of policies to complement
privatization; of particular importance is the need to set up an appropriate regulatory
and institutional framework for the post-privatization period.
Indeed, several papers have shown how a strong and independent regulatory in-
stitution can help address the negative impact of corruption on the privatization pro-
cess. Wren-Lewis (2013) uses a fixed-effects estimator on a panel of 153 electric-
ity distribution firms across 18 countries in Latin America and the Caribbean from
1995 to 2007. He regresses the log of labor employed on a corruption indicator, in-
dependent regular authority dummies (including dummies for good and bad regula-
tors), and private ownership dummies and interaction terms. Wren-Lewis employs
firm fixed effects to control for time-invariant unobservables. Because each firm is
present in only one country or province, the corruption and regulation terms are
estimated based only on changes in these variables within countries/provinces. He
also includes year fixed effects to take into account time effects. Wren-Lewis shows
that greater corruption is associated with lower firm labor productivity, but this as-
sociation is reduced when an independent regulatory agency is present. However,
because of broader institutional weaknesses, developing countries face many chal-
lenges in establishing a strong regulator. One limit of this study is that there may be
important (unobserved) parts of the reform package that also impact productivity. As
such, it should not be assumed that the (observed) reform will have the same impacts
elsewhere.
Gassner and Pushak (2014) have examined the impact that the UK regulatory
model has had in developing and transition countries, and the extent to which
they have successfully followed its key features; competition, independence and ef-
ficiency of service delivery through incentive-based regulation. The authors note
that while regulatory agencies have spread rapidly, the success of the UK regulatory
model has been only partial in middle and low-income countries. They argue that
the context of developing countries, with below cost-recovery tariffs and continued

state-ownership, makes it more difficult to establish truly independent regulatory in-
stitutions.
Thus, developing countries face many regulatory challenges; they often start with
important operational inefficiencies and insufficient revenue generation. In addition,
a majority of firms in potentially regulated sectors are still publicly-owned because

they are not attractive enough for private sector investors, and because governments
do not want to cede control of essential services. Under these circumstances, incen-
tive regulation for efficiency savings is difficult: given the low tariffs, not enough in-
vestment can be undertaken to improve service delivery, and without private profit
motives there is not a strong incentive for managers to bring about efficiency. Under-
pricing and poor operational performance are serious problems: according to the
2010 Africa Infrastructure Report published by the World Bank (2010), the under-
pricing of electricity costs the sector at least $2.2 billion a year in forgone revenues
(0.9% of GDP on average).
Recently, the concept of hybrid regulatory models has been proposed as a solution
to the challenges in developing countries (Eberhard, 2007). In hybrid models, regu-
latory contracts and independent regulatory agencies coexist. In a context where the
institutional capacity is low and/or regulatory commitment is weak, an independent
regulatory agency is supplemented by contracting out or outsourcing certain regula-
tory functions. An illustration of this is the 20-year water and electricity concession
contract in Gabon, which requires external experts to monitor the service provider’s
performance in achieving coverage targets. The experts are paid from dedicated funds
set aside from the concessionaire’s revenues and produce only nonbinding studies.
This monitoring mechanism is aimed at strengthening the independence and com-
petence of the ministerial department responsible for supervising the contract. Pol-
icymakers may also obtain regulatory assistance from regional regulators or from
other countries through twinning arrangements. For example, the Eastern Caribbean
Telecommunications Authority (ECTEL) serves the member countries of the Organ-
isation of Eastern Caribbean States as a shared regulatory body (Tremolet, Shukla,
and Venton 2004).
Taking into consideration local management and incentives is also important for
successful privatization. Liu, Sun, and Woo (2006) identify the motives of local gov-
ernment leaders and the constraints that they face during a privatization process.
These authors conclude that local governments’ motivation to privatize their SOEs
depends on whether the ownership transfer sufficiently stimulates the growth of lo-
cal tax revenues without sacrificing bureaucrats private control benefits. In addition,
Dinc and Gupta (2011) in their study of privatization in India observed that no firm
located in the home state of the minister in charge is ever privatized, which highlights
the importance of local political factors in the privatization process.
What about Remaining SOEs?
To a certain extent, the recommendations about regulation and managerial incen-

tives also apply to remaining SOEs. In fact, Bartel and Harrison (2005) argue that
public-sector inefficiency is due to the softness of budget constraints and the degree

of internal and external competition. This implies that efficiency gains in SOEs could
be achieved by reducing or eliminating government financing for public enterprises,
and/or increasing import competition.
Regarding agency-type problems, Hsieh and Song (2015) observed that one of the
key reorganizations of state-owned “corporatized” firms in China was that the par-
ent company (the controlling shareholder) of the firm incorporated as Limited Liabil-
ity Corporation was to monitor the firm and be responsible for the compensation of
the firm’s senior managers. These managers were held accountable for the firm’s bot-
tom line, which reduced agency-type problems. The senior executives of the parent
company, in turn, were directly appointed by the local government or by the Central
Organization Department of the Communist Party.
Privatization to Foreign Owners

Work on transition economies established that when SOEs are privatized to foreign in-
vestors, the efficiency gains are particularly pronounced. The results on foreign own-
ership do seem, however, to be replicated in the developing economy context. Thus
Du, Harrison, and Jefferson (2014) have found that foreign equity participation is
associated with an improvement in productivity which is greater for SOEs than for
non-SOEs in China’s manufacturing sector, suggesting that foreign firms can play an
important role in improving SOE performance. The benefits of privatization via trans-
fer to foreign firms have also been observed in the case of banking in Africa (see Clarke,
Cull, and Shirley 2005).
Part of the reason that foreign ownership improves productivity can be found in
the relation between foreign ownership and corporate risk-taking. Boubakri, Cosset,
and Saffar (2013) found that foreign (state) ownership is positively (negatively) re-
lated to corporate risk-taking, and that this relation is stronger in countries with bet-
ter institutions. To the extent that corporate risk-taking is an important driver of eco-
nomic growth, privatization via the transfer of ownership to foreign owners should
yield important economic benefits through a reorganization of prevailing incentive
structures and changes in the degree of risk aversion. Jaslowitzer, Megginson, and
Rapp (2016) also observe that risk aversion and financial conservatism are one of the
reasons that state ownership is associated with inefficiency. Using a matched panel
of 624 firms, these authors find that state ownership curtails firms’ responsiveness
to investment opportunities. Despite these findings, in some developing countries the

sale of state assets to foreigners, which carries overtones of colonial legacies, can be
a politically charged subject.
Concluding Comments

Privatization involves the transfer of productive assets from the state to private hands.
Such transfers are, by their very nature, politically sensitive and subject to potential
corruption and abuse. We outline below some important issues that policy makers in
a developing country should consider when examining a proposed privatization. In so
doing, we assume that the primary purpose of privatization is to enhance economic
growth.
First, policy-makers need to examine and establish the preconditions for success, in
terms of the business environment for competition, governance, and entry. The evi-
dence suggests that privatization has greater benefits on firm performance in stronger
business environments because the success of the process relies on effective corporate
governance of the privatized entity, as well as effective market competition. Key issues
at the national and sectoral level include:
• Depth and liquidity of the capital market (particularly important for privatization via IPO).
• Barriers to new domestic firm entry (formal entry costs, bureaucratic costs, possibilities for
incumbents to restrict entry by the use of political relationships).
• Quality of the legal system concerning corporate governance, for example company ac-
counting procedures, rules on minority shareholders, etc.
• Quality of business support, for example, legal firms, accounting firms, management con-
sultants, recruitment firms.
• Openness to foreign direct investment, both via acquisitions (via privatization) or via green-
field (to create competition), and access to foreign portfolio capital.
• Depth and competitiveness of managerial market (pool of qualified managers).
• Strength and effectiveness of competition, and competition agency.
• Independence of anti-monopoly agency from state.
The quality and independence of the state’s administrative apparatus is particu-

larly important. Privatization makes considerable demands on the capability of the
state, both in ensuring that the process is not captured by local elites, and in man-
aging the relationship between the government and the firm at arm’s-length post-
privatization, for example, via regulation. Successful privatization requires competent
government with low levels of corruption.
Turning to the privatization process itself, there is strong evidence that openness
of bidding to all, including foreign firms, is a key factor of success.
Policy-makers also need to determine the appropriate privatization methods. Re-
lated to this, the pricing of the assets to be privatized are a crucial issue with respect
to the transfer of assets from public to private hands, and the likely impact on the
distribution of income and wealth. The chosen methods depend in part on the pre-
conditions noted above. Countries with poorly developed capital markets are unlikely
to be able to privatize through IPOs. The main methods of privatization, listed on the
basis of the evidence of the literature in order of likely favorable impact on economic
growth and development are as follows:
• Sale to high-quality foreign firms.15

• Sale on domestic capital market via IPO.
• Sale to domestic businesses or business groups (trade sale).

• Sale to existing managers and/or workers.
• Free distribution of shares to the population (mass privatization).
There are obvious trade-offs. Free distribution ensures equality in the allocation
of assets around the population, but is likely to lead to weak corporate governance.
Selling to foreign owners, with appropriate safeguards, can raise company efficiency
but may lead to job losses.
Privatization seeks to improve company efficiency via corporate governance. How-
ever, as we have seen, a number of side-effects may impact other key policy targets and
these need to be considered in advance.
Social and Economic Side Effects. Higher efficiency/profitability may be obtained
through lower levels of employment, lower wages, reduced public service provision
and higher product prices, with negative distributional and social effects.
Competition Side Effects. Especially if the government is concerned with selling to
foreigners and/or maximizing revenues, competition effects may be negative and se-
rious.
Global Impact. Selling key assets such as banks or resource companies to foreign
firms may restrict the range of domestic policy and hinder long-term development.
Political Side Effects. Selling assets to elites may concentrate political power and eco-
nomic wealth into fewer hands.
Effects on Distribution of Income. An enhanced focus on the profitability of firms
may lead to increased prices of important products for poor households, as well as
reduced pay, worse employment conditions, and fewer job prospects.
Effects on Fiscal Balance. In principle, this should be unchanged because if the as-
set is priced correctly, the price should reflect the future expected earnings from the
company. In practice, pricing may be set low to achieve distributional targets or to
support elites and friends. This would worsen the government’s balance sheet. At the
same time, the new owners may be more productive than the state, and hence raise
activity and profits, with a positive effect on GDP and government revenues.
Notes
Saul Estrin is a professor of management at the London School of Economics; correspondence to be sent
to s.estrin@lse.ac.uk. Adeline Pelletier is a lecturer in strategy at the Institute of Management Studies,
Goldsmiths College, University of London. This work was supported by the U.K. Department for Inter-
national Development and the Overseas Development Institute. The authors would like to thank Tim

Green, Alberto Lemma Deborah McGurk, Anne McKinnon, Bill Megginson, John Nellis, Jon Stern, and
Jan Svejnar.
1. Kikeri and Nellis (2004) have also conducted a wide-ranging assessment of privatization.
2. Each of these three regions representing between 3% and 5% of total world privatization proceeds
over the 1988 to 2008 period.
3. The privatization barometer database provides world aggregate data on privatization and a coun-

try breakdown for developed countries. We are not aware of an alternative database providing such in-
formation. This was also confirmed by several academic and practitioner experts on privatization whom
we contacted during the course of this research.
4. Infrastructure includes transportation, water and sewerage, telecommunications, natural gas
transmission and distribution, and electricity generation, transmission, and distribution.
5. The manufacturing and services sector includes agribusiness, cement, chemicals, construction,
steel, hotels, tourism, airlines, maritime services and other sub-sectors that are not infrastructure or
finance related.
6. The financial sector includes banks, insurance, real estate, and other financial services.
7. The energy sector includes the exploration, extraction, and refinement of hydrocarbons, oil, and
natural gas.
8. The primary sector includes the extraction, refinement, and sale of primary minerals and metals
such as coal and iron ore.
9. The ownership pattern resulting from privatization often depends on the mode of privatiza-
tion chosen. Thus, private sales usually lead to concentrated strategic owners, while mass privati-
zation usually generates widespread ownership, at least initially. The impact of mode of privatiza-
tion on national economic performance in transition economies is explored in Bennett, Estrin, and
Urga (2007).
10. Note, however, that in the utilities sector (particularly for water), the technology and the na-
ture of the product restrict the possibility of competition in the market and therefore the efficiency gains
following privatization. In this case, competition for the market (to win the contract or concession agree-
ment) has to be organized. Given the ambiguous results of privatization in noncompetitive markets in
terms of improving economic performance (Megginson and Netter 2001), regulation may prove to be
more effective (Kirkpatrick, Parker, and Zhang 2006).
11. The performance of privatized banks in the seven countries of the West African Economic and
Monetary Union from 1990 to 1997 improved in the first year after privatization, but not after that.
12. Improvements in performance in Nigeria were observed in fully-divested banks, but not in the
ones where the government retained minority shareholdings.
13. Whereas competition is feasible in telecommunications markets, it is usually cost-inefficient in
the market for water services, given the scale of the investment in network assets required to deliver the
product.
14. Privatization is also not associated with the profitability and efficiency of government-owned
firms.
15. Note, however, that this method may suffer from a trade-off with competition objec-
tives since foreign firms may seek local monopoly power. Such sales may be accompanied
by conditions with respect to technology transfer, domestic content of inputs, employment,
environment, etc.
References
Acemoglu, D., and J. Robinson. 2012. Why Nations Fail. New York: Random House.
Austin, K., C. Descisciolo, and L. Samuelsen. 2016. “The Failures of Privatization: A Comparative In-
vestigation of Tuberculosis Rates and the Structure of Healthcare in Less-Developed Nations.” World
Development 78: 450–60.
Azam, J.-P., B. Biais, and M. Dia. 2004. “Privatization versus Regulation in Developing Economies: The
Case of West African Banks.” Journal of African Economies 13 (3): 361–94.
Balza, L., R. Jimenez, and J. Mercado. 2013. “Privatization, Institutional Reform, and Performance in the
Latin American Electricity Sector.” Technical Note IDB-TN-599, Washington, DC: Inter-American
Development Bank.
Barja, G., and M. Urquiola. 2001. “Capitalization, Regulation and the Poor: Access to Basic Services in

Bolivia.” WIDER Discussion Paper No. 2001/34, Tokyo: United Nations University.
Bartel, A., and A. Harrison. 2005. “Ownership versus Environment: Disentangling the Sources of
Public-Sector Inefficiency.” The Review of Economics and Statistics 87 (1): 135–47.
Beck, T., J. M. Crivelli, and W. Summerhill. 2005. “State Bank Transformation in Brazil - Choices and
Consequences.” Journal of Banking and Finance 29 (8-9): 2223–57.
Beck, T., R. Cull, and A. Jerome. 2005. “Bank Privatization and Performance: Empirical Evidence from
Nigeria.” Journal of Banking and Finance 29 (8-9): 2355–79.
Bennell, P. 1997. “Privatization in Sub-Saharan Africa: Progress and Prospects during the 1990s.” World
Development 25 (11): 1785–803.
Bennett, J., S. Estrin, J. Maw, and G. Urga. 2007. “Methods of Privatization and Economic Growth in
Transition Economies.” Economics of Transition 15 (4): 661–83.
Birdsall, N., and J. Nellis. 2003. “Winners and Losers: Assessing the Distributional Impact of Privatiza-
tion.” World Development 31 (10): 1617–33.
Boardman, A., and A. Vining. 1989. “Ownership and Performance in Competitive Environments: A
Comparison of the Performance of Private, Mixed, and State-Owned Enterprises.” Journal of Law
Economics 32 (1): 1–33.
Bonin, J. P., I. Hasan, and P. Wachtel. 2005. “Bank Performance, Efficiency and Ownership in Transition
Countries.” Journal of Banking and Finance 29: 31–53.
Bortolotti, B., and V. Milella. 2008. “Privatization in Western Europe: Stylized Facts, Outcomes, and
Open Issues.” In Privatization: Successes and Failures, edited by G. Roland, 32–75. New York: Columbia
University Press.
Boubakri, N., J.-C. Cosset, K. Fischer, and O. Guedhami. 2005. “Privatization and Bank Performance in
Developing Countries.” Journal of Banking and Finance 29: 2015–41.
Boubakri, N., J.-C. Cosset, and W. Saffar. 2013. “The Role of State and Foreign Owners in Corporate
Risk-taking: Evidence from Privatization.” Journal of Financial Economics 108 (3): 641–58.
Boycko, M., A. Shleifer, and R. W. Vishny. 1995. Privatizing Russia. Cambridge, MA: MIT Press.
Brandt, L., and G. T. Rawski. 2008. China’s Great Transformation. Cambridge: Cambridge University Press.
Chisari, O., A. Estache, and C. Romero. 1999. “Winners and Losers from the Privatization and Regulation
of Utilities: Lessons from a General Equilibrium Model of Argentina.” World Bank Economic Review
13 (2): 357–78.
Chong, A., and F. Lopez-de-Silanes, 2002. “Privatization and Labor Force Restructuring around the
World.” Policy Research Working Paper No. 2884, Washington, DC: World Bank.
Clarke, G., K. Kosec, and S. Wallsten, 2004. “Has Private Participation in Water and Sewerage Im-
proved Coverage? Empirical Evidence from Latin America.” Policy Research Working Paper No. 3445,
Clarke, G., R. Cull, and M. Shirley. 2005. “Bank Privatization in Developing Countries: A Summary of
Lessons and Findings.” Journal of Banking and Finance 29: 1905–30.
Clarke, G., R. Cull, and M. Fuchs. 2009. “Bank Privatization in Sub-Saharan Africa: The Case of Uganda
Commercial Bank.” World Development 37 (9): 1506–21.
Cull, R., and C. Spreng. 2011. “Pursuing Efficiency While Maintaining Outreach: Bank Privatization in
Tanzania.” Journal of Development Economics 94 (2): 254–61.

Davis, J., R. Ossowski, T. Richardson, and S. Barnett. 2000. “Fiscal and Macroeconomic Impact of Pri-
vatization.” International Monetary Fund Occasional Paper No. 194, IMF, Washington, DC.
Delfino, J. A., and A. A. Casarin. 2001. “The Reform of the Utilities Sector in Argentina.” WIDER Dis-
cussion Paper No. 2001/74, Tokyo: United Nations University.
Dinc, S., and N. Gupta. 2011. “The Decision to Privatise: Finance and Politics.” The Journal of Finance
LXVI (1): 241–69.

Du, L., A. Harrison, and G. Jefferson. 2014. “FDI Spillovers and Industrial Policy: The Role of Tariffs and
Tax Holidays.” World Development 64: 366–83.
Djankov, S., and P. Murrell. 2002. “Enterprise Restructuring in Transition: A Qualitative Survey.” Journal
of Economic Literature 40 (3): 739–93.
D’Souza, J., and W. Megginson, 2000. “Sources of Performance Improvement in Privatised Firms: A
Clinical Study of the Global Telecommunications Industry.” Working Paper, University of Oklahoma,
Norman, OK.
Eberhard, A. 2007. Infrastructure Regulation in Developing Countries: An Exploration of Hybrid and Transi-
tional Models. Washington, DC: World Bank.
Ehrlich, I., G. Gallais-Hamonno, Z. Liu, and R. Lutter. 1994. “Productivity Growth and Firm Ownership:
An Empirical Investigation.” Journal of Political Economy 102 (5): 1006–38.
Ennis, H. M., and S. M. Pinto. 2002. “Privatization and Income Distribution in Argentina.” Unpublished
paper delivered to a seminar on Privatization and Inequality in Latin America, Inter-American De-
velopment Bank and Universidad de las Americas, Puebla, Mexico, May.
Estache, A., V. Foster, and Q. Wodon. 2002. Accounting for Poverty in Infrastructure Reform. Washington,
DC: World Bank.
Estache, A., and M. Rossi. 2002. “How Different Is the Efficiency of Public and Private Water Companies
in Asia?.” World Bank Economic Review 16 (1): 139–47.
Estache, A., and L. Trujillo. 2008. “Privatization in Latin America: The Good, the Ugly and the Unfair.”
In Privatization: Successes and Failures, edited by G. Roland, 136–169. New York: Columbia University
Press.
Estrin, S. 2002. “Competition and Corporate Governance in Transition.” Journal of Economic Perspectives
16 (1): 101–24.
Estrin, S., J. Hanousek, E. Kocenda, and J. Svejnar. 2009. “The Effects of Privatization and Ownership in
Transition Economies.” Journal of Economic Literature 47 (3): 699–728.
Fortune. 1983. “The International 500.” August 22, 1983.
Frydman, R., C. Gray, M. Hessel, and A. Rapaczynski. 1999. “When Does Privatization Work? The Im-
pact of Private Ownership on Corporate Performance in Transition Economies.” Quarterly Journal of
Economics 114 (4): 1153–91.
Galal, A., L. Jones, P. Tandon, and I. Vogelsang. 1994. Welfare Consequences of Selling Public Enterprises.
Oxford: Oxford University Press.
Gasmi, F., A. A. Maingard, P. P. Noumba Um, and L. R. Virto. 2013. “The Privatization of the Fixed-Line
Telecommunications Operator in OECD, Latin America, Asia, and Africa: One Size Does Not Fit All.”
World Development 45: 189–208.
Gassner, K., and N. Pushak. 2014. “30 years of British Utility Regulation: Developing Country Experi-
ence and Outlook.” Utilities Policy 31: 44–51.
Gupta, N. 2008. “Privatization in South Asia.” In Privatization: Successes and Failures, edited by G.
Roland, 170–198. New York: Columbia University Press.
Gupta, N. 2009. “From Commanding Heights to Family Silver: The Halting Progress of Privatization in
India.” In Privatization Barometer Report 2009.
Gupta, N. 2011. “Selling the Family Silver to Pay the Grocer’s Bill?” Working Paper, Indiana University
https://kelley.iu.edu/nagupta/gupta_mar2011.pdf .
Gupta, N., J. C. Ham, and J. Svejnar. 2008. “Priorities and Sequencing in Privatization: Evidence from
Czech Firm Panel Data.” European Economic Review 52: 183–208.
Hailu, D., R. Guerreiro Osorio, and R. Tsukada. 2012. “Privatization and Renationalisation: What Went
Wrong in Bolivia’s Water Sector?” World Development 40 (12): 2564–77.

Hsieh, C.-T., and M. Song. 2015. “Grasp the Large, Let Go of the Small: The Transformation of the State
Sector in China.” NBER Working Paper No. 21006, National Bureau of Economic Research, Cam-
bridge, MA.
IMF. 2010. Country Report No. 10/195. July 2010.
Jaslowitzer, P., W. L. Megginson, and M. S. Rapp. 2016. “Disentangling the Effect of State Ownership on
Investment - Evidence from Europe.” Working Paper, University of Oklahoma.
Jomo, K. S. 2008. “A Critical Review of the Evolving Privatization Debate.” In Privatization: Successes and
Failures, edited by G. Roland, 199–212. New York: Columbia University Press.
Kikeri, S., and J. Nellis. 2004. “An Assessment of Privatization.” World Bank Research Observer 19 (1):
87–118.
Kirkpatrick, C., D. Parker, and Y.-F. Zhang. 2006. “An Empirical Analysis of State and Private-Sector
Provision of Water Services in Africa.” World Bank Economic Review 20 (1): 143–63.
La Porta, R., and F. Lopez-de-Silanes. 1999. “Benefits of Privatization-Evidence from Mexico.” Quarterly
Journal of Economics 114 (4): 1193–242.
Lee, J. 2009. State Owned Enterprises in China. Paris: OECD Working Group on Privatisation and Corporate
Governance of State Owned Assets.
Liu, G., P. Sun, and W. T. Woo. 2006. “The Political Economy of Chinese-Style Privatization: Motives and
Constraints.” World Development 34 (12): 2016–33.
Li, B., W. L. Megginson, Z. Shen, and Q. Sun. 2016. “Do Share Issue Privatizations Really Improve Firm
Performance in China?” Working Paper, University of Oklahoma.
Lopez-Calva, L. F., and J. Rosellon. 2002. “Privatization and Inequality in Mexico.” Unpublished paper
delivered to a seminar on Privatization and Inequality in Latin America, Inter-American Develop-
ment Bank and Universidad de las Americas, Puebla, Mexico, May.
Lopez-de-Silanes, F. 2005. “Dos and Don’ts in Privatization: Evidence from Less Developed Countries.”
Privatization Barometer Newsletter No. 2: 29–35.
Macquieira, C., and Z. Salvador. 1996. “Privatisaciones en Chile: Eficiencia y Politicas Financieras.” Es-
tudios de Administracion 3 (2): 1–36.
Majumdar, S. K. 1996. “Assessing Comparative Efficiency of the State-Owned, Mixed, and Private Sec-
tors in Indian Industry.” Public Choice 96 (1/2): 1–24.
Maquieira, C., and S. Zurita 1996. “Privatization in Chile: Efficiency and Financial Policies.” Administra-
tion Studies 3 (2): 1–36.
Megginson, W. 2000. “Privatization.” Foreign Policy 118: 14–27.
Megginson, W. L. 2005. “The Economics of Bank Privatization.” Journal of Banking and Finance 29 (8-9):
1931–80.
Megginson, W., R. Nash, and M. van Randenborgh. 1994. “The Financial and Operating Performance
of Newly Privatised Firms: An International Empirical Analysis.” Journal of Finance 49 (2): 403–52.
Megginson, W., and J. Netter. 2001. “From State to Market: A Survey of Empirical Studies on Privatiza-
tion.” Journal of Economic Literature 39 (2): 321–89.
Megginson, W. L., and N. Sutter. 2006. “Privatization in Developing Countries.” Corporate Governance
14 (4): 234–65.

Megginson, W. 2014. “Privatization Trends and Major Deals in 2013 and 2014.” In Privatization
Barometer Report 2013–2014.
Nellis, J. 2008. “Privatization in Africa. What Has Happened? What Is to Be Done?” In Privatization: Suc-
cesses and Failures, edited by G. Roland, 109–135. New York: Columbia University Press.
Otchere, I. 2005. “Do Privatised Banks in Middle- and Low-income Countries Perform Better than
Rival Banks? An Intra-Industry Analysis of Bank Privatization.” Journal of Banking and Finance

29(s 8–9):2067–93.
Piketty, T. 2014. Capital in the Twenty-First Century. Cambridge, MA: Belknap Press.
Privatization Barometer. 2015. “Privatization Barometer Report 2014–2015.” The wave builds.
Roland, G. 2008. “Private and Public Ownership in Economic Theory.” In Privatization: Successes and
Failures, edited by G. Roland, 9–31. New York: Columbia University Press
Tan, J. 2012. “The Pitfalls of Water Privatization: Failure and Reform in Malaysia.” World Development
40 (12): 2552–63.
Tian, G. 2000. “State Shareholding and Corporate Performance: A Study of a Unique Chinese Data Set.”
Working Paper, London: London Business School.
Torero, M., and A. Pasco-Font. 2001. “The Social Impact of Privatization and the Regulation of Utilities
in Peru.” WIDER Discussion Paper No. 2001/17, Tokyo: United Nations University.
Tremolet, S., P. Shukla, and C. Venton. 2004. Contracting Out Utility Regulatory Functions. Washington
DC: World Bank.
Verbrugge, J., W. Owens, and W. Megginson. 2000. “State Ownership and the Financial Performance of
Privatised Banks: An Empirical Analysis, Proceedings of a Policy Research Workshop at the World
Bank, March 15–16, 1999.” Dallas: Federal Reserve Bank Dallas.
Vickers, J., and G. Yarrow. 1988. Privatization: An Economic Analysis. Cambridge, MA: MIT Press
Wallsten, S. 2001. “An Econometric Analysis of Telecom Competition, Privatization, and Regulation in
Africa and Latin America.” Journal of Industrial Economics 49 (1): 1–19.
World Bank 2010. Africa’s Infrastructure: A Time for Transformation. World Bank: Washington D.C.
Wren-Lewis, L. 2013. “Do Infrastructure Reforms Reduce the Effect of Corruption? Theory and Evidence
from Latin America and the Caribbean.” Policy Research Working Paper No. 6559, Washington, DC:
World Bank.
Zhang, Y.-F., D. Parker, and C. Kirkpatrick. 2008. “Electricity Sector Reform in Developing Countries:
An Econometric Assessment of the Effects of Privatization, Competition and Regulation.” Journal of
Regulatory Economics 33 (2): 159–78.
Public-Private Partnerships in Developing
Countries: The Emerging Evidence-based

Critique
James Leigland
Advocates of public-private partnerships (PPPs) for infrastructure services in developing

countries have long battled criticism of these arrangements by civil society groups. The
view among PPP advocates generally has been that these criticisms are mostly ideological
polemics that mix opinion with selected but often misinterpreted facts. But over the last two
decades, as the experience with PPPs has increased in both developed and developing coun-
tries, a different kind of critique has emerged, one that is based on non-ideological empirical
research, and is sometimes expressed by PPP advocates. These studies often focus on individ-
ual aspects of PPPs, and usually do not claim to be “PPP evaluations” or express opinions
on the overall value of PPPs. Taken together, a powerful, evidence-based critique of PPPs is
emerging, but one that is more measured than much of the criticism of the last two decades.
This new critique recognizes many cases in which PPPs have not been successful, but also
some situations in which PPPs can generate value for money. Because of its critical tone,
some of this research is now regularly cited by the civil society critics of PPPs, giving their
arguments more weight than was the case a decade ago. This paper attempts to summarize
some of the most compelling examples of this kind of emerging critique, and uses the sum-
mary to assess the practicality of the G20’s recent advocacy of large, “transformational”
PPPs as tools for dealing effectively with infrastructure challenges in low-income countries.
Global support for public-private partnerships (PPPs) for infrastructure seems

stronger than ever before. Discussions in G20 meetings over the last several years
have increasingly focused on the need for a huge scale-up in infrastructure investment
in developing countries, particularly low-income countries. G20 pronouncements
talk about the advantages of realizing this scale-up via large, “transformational”
projects involving private sector participation. By this they mean large, regional, or
doi: 10.1093/wbro/lkx008 33:103–134
cross-border infrastructure projects involving private investment and management,
which potentially have positive, transformational impacts on entire countries or re-
gions. From time to time over this period, the G20 has considered efforts to help mod-
ify the mandates of national and international development banks so that these in-
stitutions will take the lead on such PPPs and crowd in the private sector.

As a result, several development banks have been considering adjustments to their
business models to give more attention to regional infrastructure PPPs. During the
negotiations in 2013 for the 17th replenishment of the International Develop Asso-
ciation (IDA17), the World Bank proposed using IDA funding to help develop trans-
formational PPP projects. Those proposals have now evolved into the Bank’s design
of the Global Infrastructure Facility (GIF), an entity meant to coordinate the efforts of
MDBs, private investors, and governments to prepare and structure PPPs. The BRICS
countries, at their summit in Durban in March 2013, announced plans to create a
new development bank (now known as the New Development Bank) that would focus
on infrastructure, and do so in a way that would make up for the deficiencies of the ex-
isting international financial architecture and help catalyze the private sector invest-
ment needed in rapidly-growing BRICS economies (Republic of South Africa 2014).
Perhaps the most ambitious and concrete commitment of this kind to date is the de-
cision, announced by the African Development Bank (AfDB) in July 2013, to create
a billion-dollar preparation and financing facility for large infrastructure projects in
Africa, referred to as Africa50. The institution’s purpose “is to unlock private financ-
ing sources... and to accelerate the speed of infrastructure delivery in Africa,” (AfDB
2013).
But as this enthusiasm for PPPs is growing, so is a less widely-recognized body of
research that takes a much more measured approach; it still represents a kind of ad-
vocacy, but one that incorporates a greater degree of critical analysis of PPP suc-
cesses and failures. A recent example of this is evident in a World Bank working pa-
per by Michael Klein (2015), an influential PPP advocate during the 1990s and early
2000s. Klein notes that despite more than two decades of use and refinement of the
PPP mechanism, there are still no consistent geographical patterns of usage: “The
general picture is one of waves of enthusiasm for PPPs followed by some disenchant-
ment and consolidation. Different countries were caught up in the waves at different
times.” What accounts for this lack of sustained enthusiasm? Klein says that eval-
uations show that PPPs can outperform public sector firms, and “are useful tools for
reform of service delivery” (Klein 2015). But it is no longer clear that PPPs are consis-
tently better run than public firms. “The evidence suggests that well-run public firms
tend to match the performance of private firms in regulated sectors” (Klein 2015).
Klein’s comments are a reminder that a significant amount of evidence-based re-
search on PPPs has accumulated since the late 1990s. But a good deal of it, partic-
ularly over the last decade, has not been uniformly positive about PPPs, at least not
in the fashion of the largely promotional literature published by MDBs and donors in
the 1990s and early 2000s. Some of this research, for example by economist Antonio
Estache, is now being used to bolster PPP criticisms prepared by civil society groups
(e.g., Alexander 2013). Such groups have produced a broad collection of critical
PPP studies: International Rivers (Bosshard, 2012); Public Services International
(Hall 2015); Heinrich Boell Foundation (Alexander 2013); CEE Bankwatch Network

(2008); Oxfam (Marriott 2014); the Bretton Woods Project (2016). These groups
have long been critical of PPPs, but in the past their arguments against private par-
ticipation have often seemed more ideological than evidence-based, and therefore not
very compelling. But the growing use of evidence-based research reported on by re-
spected social scientists like Estache, Klein, and others has added weight to their ar-
guments, warranting more careful consideration by PPP advocates.
Much of this evidence-based critique of PPPs is dispersed in collections of working
papers and academic articles, or focuses on individual aspects of PPP projects, and
does not claim to be “PPP evaluations.” In any case, this body of research is having
no noticeable impact on PPP discussions in organizations like the G20. This paper at-
tempts to summarize some of the most compelling examples of this kind of emerging
critique by organizing them into several key themes: (i) how prevalent is the usage of
PPPs in the developing world; (ii) the costs and profits associated with these projects;
(iii) the significant outlay involved in preparing these projects; (iv) their struggle to at-
tract commercial financing; (v) how developmental they are; and (vi) the institutional
and political problems that limit success. The discussion of each key theme also at-
tempts to extract insights about some of the efforts to deal with these criticisms and
establish conditions under which PPPs are likely to provide value for money. Finally,
this paper uses this summary to (vii) assess the practicality of the G20’s recent pro-
motion of “transformational” PPPs as mechanisms for dealing effectively with infras-
tructure challenges in low-income countries.
As there is no universally accepted definition of “public-private partnership”
(PPP), this paper adopts Klein’s broad definition of PPP to mean “private participa-
tion” in infrastructure sectors like telecoms, transport, water and sanitation, waste
management, and electricity: “Activities that fall under this umbrella may sometimes
be characterized, for example, as ‘concession’ or ‘franchise’ or ‘build-operate-transfer’
deals” (Klein 2015). Where indicated, some of the data cited here covers “private
participation in infrastructure,” which includes information on privatized utilities,
as well as more traditional kinds of infrastructure PPPs.
PPP Prevalence
The foundation of Klein’s argument is that although many countries use PPPs at least
occasionally, the prevalence of usage surges in waves, often driven by fiscal problems
or other ways in which the public system has “run into trouble,” (Klein 2015). But
the waves inevitably recede, “... in seemingly random patterns,” (Klein 2015). Klein
Leigland 105
Table 1. Sources of Annual Financial Flows to SSA Infrastructure, in US$ billions
Operations &
Maintenance Capital Investment All Spending
Official
Public Sector Public Sector Non-OECD Development Private

O&M Capex Financiers Assistance House-holds Private Sector Total Capex All Sources
20.4 9.3 2.4 3.7 2.1 7.3 24.8 45.2
Percent of 38% 10% 15% 8% 29% 100%
Total Capex
Source: Foster and Briceño-Garmendia 2010.
attributes the shallowness of PPP popularity to a lack of clear and consistent evidence
that PPPs perform better than public sector organizations. He estimates that as of
2015, while PPPs account for a share of the total infrastructure investment in low-
and especially middle-income countries, it is normal for a country to use PPPs for only
about 15 to 25 percent of total infrastructure investment.
Other sources of data suggest even lower levels of PPP prevalence. The World
Bank’s Africa Infrastructure Country Diagnostic study (AICD), published in 2010,
found that in total, the private sector accounted for an impressive level of infrastruc-
ture investment in sub-Saharan Africa (SSA) by contributing about 29 percent of
total capital spending (table 1).
But the AICD further qualified this data in several important ways: First, it demon-
strated that private investment was heavily skewed in terms of countries, with
about 60 percent of total SSA private sector investment shared equally by just two
countries—Nigeria and South Africa. Second, private investment was heavily skewed
in terms of sectors, with 77 percent of SSA’s private investment since 2000 going to
telecommunications, mostly via build-own-operate projects (BOO). According to the
AICD, the energy sector, which is arguably the most in need of urgent major capital
investment, attracted only 10 percent of total private investment.
Other studies tend to support lower usage figures. Burger and Hawkesworth
(2011) surveyed 22 countries (19 OECD countries and three middle-income coun-
tries) regarding value-for-money issues associated with PPPs. Of these, eighteen
countries reported that less than 10 percent of public sector infrastructure invest-
ment took place via PPP arrangements. From 2000 to 2010, the UK’s Private Finance
Initiative (PFI) probably averaged a higher annual percentage of total infrastructure
investments via PPPs than most other OECD countries, at about 12 percent. The only
two non-OECD countries surveyed, Mexico and Chile, reported that over 20 percent
of their infrastructure investment occurred via PPPs. A number of the countries sur-
veyed admitted informally that they did not foresee PPPs exceeding 15 percent of total
public investment (Burger and Hawkesworth 2011).
Figure 1. PPI Investment by IDA Status, 1995 to 2015, in $ millions

Source: World Bank PPI Database as of Jan. 2017
Across the developing world, PPPs play a relatively small role in infrastructure in-
vestment, averaging between 15 to 20 percent according to the Independent Evalua-
tion Group of the World Bank (Independent Evaluation Group 2014). In the poorest
developing countries, the use of PPPs has been even more negligible. Figure 1 demon-
strates this, using data from the World Bank’s PPI Project Database to show invest-
ments related to “private participation in infrastructure” (PPI) in countries eligible for
support from the International Development Association (IDA; i.e., countries whose
Gross Net Income per capita is below $1,215), and contrasts these against data from
non-IDA developing countries (“blend” countries have been excluded).1 In its review
of PPI activity in IDA countries since 2011, a World Bank report remarks: “The mar-
ket for PPIs has not been expanding,” (Ruiz-Nunez 2016).
In the developing world, a share of infrastructure investment in the range of 15 to
20 percent does not mean that PPPs have failed to play a significant role in infrastruc-
ture. But it is far less than what was expected of PPPs in the 1990s when Klein and
his colleagues at the World Bank were considering sharp reductions in infrastructure
lending because they expected the private sector to eventually play a more dominant
role in bridging the gap and financing and managing infrastructure services in that
region of the world.2
What does this information about PPP prevalence tell us about the conditions
under which PPPs are likely to provide value for money? The message is simple:
PPPs work much better in middle-income economies than they do in low-income
Leigland 107
countries. This means that in most cases a complex, long-term, brownfield concession
for retail water distribution, for example, requiring significant capital investment,
should not be the first choice as the service delivery solution in a least-developed coun-
try (as such contracts often were in the early 1990s). This review suggests that the
poorest countries can usually benefit more from traditional technical assistance and

capacity building, or from hybrid projects that mix elements of PPP contracts with
those of consulting or engineering, procurement, and construction (EPC) contracts
to reduce risks for the private partners. Reforms to legal and regulatory frameworks
within which PPPs eventually would be structured are also critical in these countries,
along with help in improving government procedures for things like procurement and
construction management.
Costs and Profits

PPPs involve multiple stakeholders, all of whom have interests in how the project per-
forms. The conventional view of PPPs is that, when compared with typical public
projects, they should provide better service at lower costs. The private partner needs
to make a profit, but the return should be reasonable. Projects that do not meet these
expectations can be subject to public criticism from government, the news media, user
groups, civil society, etc. But getting the balance right in a way that satisfies all key
stakeholders is not an easy task, as the recent history of PPPs indicates.
The profitability of private firms involved in PPP projects, and the resulting costs
for government and end-users, has long been a controversial subject in industrialized
countries like the United Kingdom, where the PFI became one of the OECD’s best-
known PPP programs starting in the early 1990s. Studies of the profitability of PFI
deals in the U.K. health sector have found rates of return as high as 60 percent (Shaoul
2008).
Whether or not the cause is high profits, PPPs have often tended to be more ex-
pensive than traditional public procurement. A 2006 report by the European In-
vestment Bank (EIB) reviewed the costs of 227 road projects in 15 European coun-
tries and concluded that projects done as PPPs (65 of the total), were 24 percent
more expensive than those done via traditional public procurement (Blanc-Brude,
Goldsmith, and Välilä 2006). In a 2015 review of effective interest rates on pri-
vate finance projects, the U.K.’s National Audit Office found that these rates, at
7 to 8 percent, were double the rates on normal government borrowing, at 3 to
4 percent (U.K. National Audit Office 2015). In a 2016 review of PPP literature
sponsored by the UN’s Department of Economic and Social Affairs, the authors
concluded that “Overall, the evidence suggests that PPPs have often tended to be more
expensive than the alternative of public procurement...” (Jomo et al. 2016).
Most industrialized countries try to anticipate project benefits and costs, including
private profits, by requiring a “value for money” (VFM) analysis of PPP projects. VFM
analysis involves estimating project costs, including profits for the private partners,
and measuring them against project benefits, including service quality, quantity, and
prices for governments or end-users. Quantitative VFM assessment typically involves
comparing the chosen PPP option against a “public sector comparator” (PSC). The
PSC allows a comparison of the risk-adjusted cost to government of procuring the

project through traditional procurement (the PSC), with the expected cost to govern-
ment of the PPP (pre-procurement) or the actual PPP bids (post-procurement).
The survey by Burger and Hawkesworth (2011) found that 17 of 22 countries used
“public sector comparators” to assess the value for money of PPPs. But ever since
the technique was first refined and pioneered as part of the UK’s PFI program in the
1990s, it has been criticized for being inaccurate and subject to manipulation, lead-
ing some observers to conclude that it is often an expensive way of endorsing the
pre-selected choice of private participation. A UK Audit Commission report (2003)
concluded that “the PSC has lost the confidence of many people, and risks being seen
more as a hoop to jump through on the way to government funding than a valuable
exercise that can help ensure better VFM.”
Engel, Fischer, and Galetovic (2014), noted that in some countries PPPs are at-
tractive to government not necessarily because they are expected to be less expensive,
but simply because accounting rules allow project costs to be moved off government
books in order to give the appearance of lower debt levels. Klein also mentions this
and notes that when the United Kingdom changed its accounting rules in 2009, PFI
projects became less attractive as a result. If true, this may account for some of the
well-documented failures of the PSC to accurately forecast PPP project costs.
Of course, other kinds of quantitative comparisons can be done on an economic
cost-benefit basis, using a wide variety of VFM methodologies (World Bank Institute
and Public-Private Infrastructure Advisory Facility 2013). It is also true that regard-
less of the method used, huge mistakes are routinely made in the estimation of finan-
cial costs and benefits associated with infrastructure projects—whether or not PPPs
are involved. In a survey of 58 rail projects (a mixture of public and private projects),
Flyvbjerg (2005) found that costs were underestimated by an average of 45 percent
and demand forecasts were overestimated by an average of 51 percent. Flyvbjerg con-
cluded that such consistently large mistakes in cost-benefit estimations must be at-
tributed to a combination of faulty techniques, as well as causal effects like “optimism
bias” and “strategic misrepresentation.” Doing an infrastructure project as a PPP
does not lead to any more accurate estimation of costs and benefits. Vassallo (2007)
and Bain (2009) investigated the forecasting performance of privately-financed toll
roads; their findings were similar to Flyvbjerg’s. These authors found that PPP toll
road traffic forecasts were typically characterized by large errors and considerable op-
timism bias.
However, in the developing world, VFM techniques like PSCs are arguably even
more unreliable than in OECD countries. Even industrial countries have little objective
Leigland 109
relevant data upon which to base cost estimates. Without such data, which is virtu-
ally non-existent in low-income countries, calculating with any accuracy how much
a project will cost over 25 to 30 years of operation is almost impossible. As noted by
Estache and Philippe (2012), officials in these countries often find out, after contracts
have been signed, that original forecasts of project costs and profits were inaccurate.

Indeed, projects often turn out to be unprofitable for private partners, triggering rene-
gotiations:
“Experience has shown that besides the short-term subsidies sometimes needed
to support privatization processes, the public sector often has eventually had to
commit subsidies for the long-term as well–usually as one of the outcomes of a
renegotiation,” (Estache and Philippe 2012).
A widely-cited study by Sirtaine et al. (2004), examined the profitability of PPP

projects in Latin America during the late 1990s using a sample that included both
brownfield and greenfield projects. The Sirtaine study suggested that on average,
projects demonstrated profitability only after about 10 years. Up to that point, project
shareholders earned negative returns on their investments, even when adding in
management fees, estimated accumulated capital gains, and potential investment
markups. The study further found that 40 percent of the sampled concessions did
not have the potential to ever become profitable, with that proportion increasing to
50 percent for concessions in the energy and transport sectors. Indeed, Sirtaine esti-
mated that over the history of these projects, on average they were unable to gener-
ate sufficient annual operating income after taxes to cover all of their financial obli-
gations. Only by adding in “indirect forms of dividends” like investment mark-ups,
transfer fees, and payments for capital appreciation paid at the end of the contract
period could these concessions generate reasonable remuneration for their private
partners. The study suggested that, at a minimum, these PPPs face constant cash
flow problems over at least the first decade of their existence.
Another study, by Guasch (2004), reached conclusions consistent with Sirtaine’s
findings. Guasch found high rates of PPP contract renegotiation after only a few years
of operation, that is, long before Sirtaine says that private partners are able to con-
firm their projects’ long-term profitability. Using a sample that also included brown-
field and greenfield projects in Latin America, the study concluded that PPI projects
in that region registered a high incidence of renegotiation, about 42 percent, coming
on average after only 2.2 years of operation. The results of renegotiation tended to
favor operators, mostly with improvements to cash flow and profitability via compro-
mises such as permitted delays in investment obligations (69 percent), reductions in
investment obligations (62 percent), tariff increases (62 percent), and increased pass-
through to tariffs of cost items (59 percent). Again, this suggested that PPP cash flow
problems tended to be severe, generating stresses that operators and investors had to
live with over the long term, and which were critical in precipitating many renegoti-
ations.
Estache and Philippe (2012) concluded that sectors like telecommunications and
electricity generation in which cost-reflective tariffs seem less controversial have
proven to be reasonably profitable and largely free of subsidies. But most other sec-

tors in the developing world—electricity distribution and transmission, transport,
and water and sanitation—often require subsidies to sustain cash flows or bring the
rate of return close to the cost of capital. If, with careful analysis the subsidies are not
built into the initial contracts, they are likely to result from renegotiation of those con-
tracts. In the latter event, renegotiation increases the risk of a less than optimal reduc-
tion in the fiscal benefits that governments care most about, that is, the size and pace
of investment. Renegotiations also tend to increase the most politically painful costs—
tariffs or off-take payments—sometimes to unaffordable levels for governments.
How are governments and their development partners coping with the fact that
PPPs are costlier and less profitable than assumed in the 1990s? One way is to rely
more heavily on “blended finance” approaches to PPPs. Since the launch in the early
2000s of the multi-donor trust fund for output-based aid (the Global Partnership on
Output-Based Aid), “blended finance” has become increasingly popular as a way of
using concessional finance to catalyze private sector investment, particularly in in-
frastructure PPPs. The International Finance Corporation’s (IFC) Blended Finance
Unit, launched in 2007, and the EU’s regional blending finance facilities, such as the
EU-Africa Infrastructure Trust Fund, have all used subsidies to bring down the costs
of various kinds of infrastructure PPPs (IFC 2012). The use of blended finance in
this way creates a hybrid approach that combines PPP elements with those of more
traditional public projects.
But blended finance involves subsidies, and the use of subsidies requires justifica-
tion to ensure that it is really crowding in private finance rather than crowding it out.
Economists typically recommend the use of cost-benefit analyses for such justifica-
tions to clearly identify any obstacles that reflect market failure and help determine
whether subsidized finance can solve the problem. Theoretically, cost-benefit analy-
ses can confirm that the likely development impacts of using subsidized finance far
outweigh the distortions that may result. When this kind of analysis can be done, it
almost certainly leads to more developmental projects. But the difficulties involved
in this should not be underestimated. The World Bank’s Independent Evaluation
Group found in 2010 that the use of cost-benefit analysis dropped from 70 percent
of all World Bank projects in 1970 to 25 percent in 2008, largely because of prob-
lems with data and the difficulties in quantifying costs and benefits (Independent
Evaluation Group 2010).
Leigland 111
Preparation Costs
The OECD (2008) has indicated another important reason for the high costs asso-
ciated with PPPs in many countries—the high cost of PPP project preparation, es-
pecially when compared with the costs of traditional public procurement. Prepara-

tion costs include the legal, financial, and technical costs incurred by both public and
private sector actors in developing a PPP for commercial operation, and so include
“transaction costs” associated with PPP procurement processes and contract negoti-
ation, as well as (especially in some developing countries) “upstream” legal, regula-
tory, and policy preparation tasks that go well beyond normal transaction costs.
In their review of the PPP literature, De Schepper, Haeqendonck, and Dooms
(2015) noted widespread agreement among practitioners and academics that PPP
preparation costs are higher than preparation costs associated with traditional pub-
lic procurement. In many cases, these costs are so high that they discourage poten-
tial bidders from competing for projects, and in some cases undermine the basic cost-
effectiveness rationale of PPPs and negatively impact on the economic and financial
viability of projects. De Schepper’s study of 172 public infrastructure projects in Bel-
gium found much longer and more complex bidding processes associated with PPPs
than with projects procured via traditional public methods.
There are several reasons why PPP preparation costs are so much higher than the
costs of preparing projects involving traditional public procurement. The survey by
Burger and Hawkesworth, which found widespread use of “public sector compara-
tors” to ascertain value for money of PPPs, also noted that the process of preparing
a PSC and requiring each bidder to prepare a projected PPP model is costly and time-
consuming. Complex PSCs can take several months to finalize, and the resulting re-
duction in net benefits for potential private partners may cause them not to bid on
projects that are deemed to be too small, with potential revenues not high enough to
compensate for preparation costs. On the other hand, some projects are avoided be-
cause they seem too large and complex, leading to expensive bid preparation and time
and cost overruns resulting from protracted contract negotiations. This may explain
the findings of Zitron (2006), which showed that on average, only three bidders com-
peted for each PPP contract in the United Kingdom, and that there were fewer than
three bidders on one-quarter of the contracts. Governments can partially or com-
pletely compensate firms for their bidding costs, but of course this increases the cost
of the PPP option relative to the traditional public procurement option.
Twelve of the countries in the Burger and Hawkesworth survey said they used
some form of cost-benefit analysis to assess value for money for projects involving
traditional public procurement, but these appear to be much less rigorous tests than
PSCs and do not constitute the public-sector equivalent of “private sector compara-
tors.” Only a few countries applied criteria to all prospective projects to establish
which mode of procurement would result in the highest value for money. In other
words, for many countries traditional public procurement remains the lower cost, less
complicated, default option for procurement, and the PPP choice depends on the dis-
cretion of departmental officials.
There is no single widely accepted metric for infrastructure project preparation
costs. Costs may be measured as a percentage of initial capital costs, construction

costs, or even the total net present value of the deal over its entire lifetime. Most pub-
lished work has focused on “transaction” costs, usually synonymous with “down-
stream transaction costs,” and mostly procurement-related costs incurred by govern-
ments or their MDB partners. A sampling of different metrics follows below:
• In a widely-cited study of the procurement phase costs of bidding and negotiating con-
tracts for 55 infrastructure PPPs in six sectors of the U.K. economy, Dudkin and Valila
(2005) concluded that total transaction costs of this type averaged over 12 percent of the
capital value of the projects, with the public-sector cost at about 3.5 percent, the winning
bidder’s cost at about 3.8 percent, and costs to losing bidders totaling about 5 percent.
• The World Bank (Lin and Doemeland 2012) estimated that MDB and government prepa-
ration costs for the Nam Theun 2 hydropower project in Lao PDR (total investment: $1.4
billion) amounted to $124 million at project close in 2005, or 9 percent of total project
investment.
• Farajian (2010) found lower average transaction costs in several case studies of large
transport PPPs in the United States, but concluded that the transport agencies involved
were not reporting all relevant costs. He also found that some individual state departments
of transportation were requiring that project managers’ budget up to 10 percent of total
investment for procurement and contract negotiations managed by state officials.
• Castalia Strategic Advisors (2010) estimated government transaction costs of PFI school
projects in Australia and New Zealand to be 10 to 11 percent of total project construction
costs, with significant premiums to be expected for first-time projects in either country.
• Bhattacharya, Romani, and Stern (2012), found that for projects in developing countries
involving limited experience with the type of project or technology, or in a low-capacity
country, preparation costs, including costs of design and arranging financing “can consti-
tute up to 10 percent of overall investment costs.”
• The AfDB (2013) estimated that their preparation costs for large infrastructure projects
in Africa can reach 10 to 15 percent of project capital costs.
These kinds of studies suggest that PPP project preparation costs in developing
countries are much higher than they are in OECD countries because of the need to
include costs for things like upstream preparation and premia for new or particularly
complicated sectors like hydropower. These costs are also considerably higher than for
projects involving traditional public procurement because of the need in many cases
to carry out value-for-money analyses (which are typically less onerous for public
procurement) and use more complex bidding processes.
Under what conditions can governments and their development partners deal ef-
fectively with the time and costs involved in project preparation? For many years,
the default solution was to make as much of this preparation as possible the re-
sponsibility of the private sector. This attitude was an outgrowth of the notion that
Leigland 113
implementing a PPP meant handing over government problems with infrastructure
to private companies for solution. Probably the first notable example of this approach
was the Buenos Aires water concession, signed in December 1992. This was one
of the first, large brownfield infrastructure concessions. A “defining feature” of the
tender process was a lack of information about the water system and its problems

(Alcazar, Abdala, and Shirley 2000). Bidders were supposed to do all of their own due
diligence, and the winning bidder was supposed to refine its implementation plan in
the first few months after signing the contract. A protracted series of contract rene-
gotiations would follow because latent defects made the project much more expensive
than anticipated.
That approach has been frequently copied over the years (e.g., Kotze, Ferguson, and
Leigland 2000), but by the early 2000s, donors and MDBs began to recognize that:
(i) private partners cannot by themselves fully prepare PPP projects in a way that
optimizes economic benefits; and (ii) a much more substantial effort by donors and
MDBs to pay for and supervise preparation, before private partners become involved
can result in more effective, sustainable, and pro-poor projects. The stronger role for
governments and their development partners in identifying problems and designing
solutions for private partners to implement is a characteristic of hybrid management
contracts being developed or implemented in countries like Benin, Liberia, and Sierra
Leone (Republic of Sierra Leone 2015). These contracts shift risks away from private
partners, toward governments, donors, and MDBs, who are, theoretically, better able
to mitigate those risks. Ultimately, this should make the contracts more productive
and sustainable.
On a broader scale, a number of multi-donor project preparation facilities were
established in the 2000s to deal with the high costs of preparation, but evaluations
later showed that these facilities were too small and too bureaucratic to have much
of a substantial impact (Cambridge Economic Policy Associates 2012). As noted in
the introduction above, much larger preparation platforms have been created more
recently to try to address the problem, but they have not yet had a chance to show
results in terms of sustainable PPPs.3
Finance and Investment

PPP advocates expect, and sometimes promise, that such contracts will lead to im-
proved efficiency and increases in investment from private sources, consequently
leading to greater service capacity and coverage. But again, the evidence suggests
that this is not consistently the case. As various studies demonstrate, private sources
of finance have not played as strong a role in infrastructure PPPs in developing coun-
tries as hoped. In Africa, the long-standing practice in some sectors of government
on-lending concessional finance to private concessionaires illustrates this fact. Many
of the rail concessions in SSA benefited from this kind of debt, sourced as relatively
inexpensive sovereign guaranteed loans from bi-lateral donors or MDBs. These con-
cessions often needed financial help because concessionaires could not or would not
borrow to finance assets like rail lines, with operational life-spans being much longer
than the terms of the concession contracts. Governments themselves often filled this
gap by providing a large share of project financing. In the 1990s, this kind of on-

lending was usually done with an interest rate mark-up to the concessionaire to cover
government administrative costs and loan loss liability, but was still lower than the
commercial rates that the concessionaire could obtain. As commercial lending for
all kinds of brownfield concessions began to diminish following the Asian Crisis, the
mark-ups were slashed to compensate for diminished global project finance lending,
and by the mid-2000s, in some cases the mark-ups disappeared entirely (Pozzo di
Borgo et al. 2006).
A study by Gassner, Popov, and Pushak (2009), examined whether water and
electricity utilities involving private participation outperformed those run by gov-
ernments. The study used a data set of more than 1,200 utilities in 71 developing
countries. The results of the study showed that the private sector delivered higher
labor productivity and operational efficiency, convincingly outperforming compara-
ble companies that remained state owned and operated. But the study found mixed
evidence regarding increases in investment and could not conclude that investment
usually increases with private participation.
Gassner addressed the investment question in terms of different sectors and types
of private participation. For electricity divestitures, investment per worker increased
with private participation, as economic theory predicts. For lease and management
contracts, which are more common for water and sanitation than for electricity,
private partners were usually not obligated to make major capital investments, but
Gassner’s results suggested that even if private participation generated operational
improvements, the public utility companies involved did not increase investment. And
for concession contracts, which also tend to generate improvements in operational
efficiency, “... there is no conclusive evidence that investment increases,” (Gassner,
Popov, and Pushak 2009). If efficiency improves, but is not supported by increases in
public or private investment in the maintenance and expansion of utility networks,
Gassner asked how operational improvements could be sustained over the long term.
A second study, by Marin (2009), undertook a global review of public-private part-
nerships for urban water utilities in developing countries. The study focused on 65
large urban water PPPs in operation for at least five years. Marin concluded that the
principal benefit of these projects was operational efficiency rather than their ability
to supply private sector finance. The author also acknowledged that private invest-
ment was the main attraction of PPPs in the water sector in the 1990s, but added
that “experience has shown that this was largely the wrong focus,” (Marin 2009).
A host of other studies have found PPP-related efficiencies generated un-
der different conditions, in different sectors, including electricity distribution,
Leigland 115
telecommunications, and water distribution (Estache and Rossi 2004; Andrés,
Schwartz, and Guasch 2013), and transport (Perelman and Serebrisky 2012).
Estache and Philippe (2012) survey many other studies showing some measure of
PPP efficiency in different sectors. But again, these studies do not consistently show
a corresponding increase in investment.

Government on-lending to PPPs accelerated in some countries after the onset of
the 2008 global financial crisis, which caused a collapse of the syndication mar-
ket, inactive bond markets, and unusually high costs of funding. Governments and
DFIs used concessional lending to meet funding shortfalls resulting from projects un-
able to raise sufficient commercial debt finance on acceptable terms. The U.K. govern-
ment attempted to salvage its PFI with a proposed investment fund that would pro-
vide up to 100 percent of the financing needed for projects to reach financial closure
(Farquharson and Encinas 2010). In developing countries such as Brazil, India, Mex-
ico, and South Africa, government-owned development finance institutions provided
funding or guarantees to kick-start PPP projects. Because countries like Brazil and
India accounted for such a large share of private investment activity in developing
countries, their use of DFIs and local public banks to fund large projects kept global
PPI market investment numbers from declining as much as expected from 2008 to
2009 (Izaguirre 2009).
The global financial markets have still not fully recovered from the financial cri-
sis and resultant increased financial regulation, at least not in the sense of returning
to earlier levels of high liquidity, substantial risk appetites, and low cost of project fi-
nance debt provided by commercial banks. Governments, especially those in larger
middle-income countries, continue to replace scarce commercial finance with public
money from government budgets or donors and MDBs. Recent efforts by the World
Bank’s PPI Database to collect detailed information on PPP financing sources in de-
veloping countries confirm that the private sector plays even less of a PPP financing
role in low-income countries. In 2015, 57 percent of PPP funding in IDA countries
came from governments, multilaterals, and bilateral institutions, with the balance
coming from commercial sources. For all developing country regions in 2015, the
percentage was 41 percent (table 2). In her analysis of this data, one World Bank
staff member noted that, “One of the prevailing notions about PPPs is that upfront
costs are wholly paid for by the private sector... However, this is a myth,” (Chao 2016).
What does this critique say about the conditions under which affordable private fi-
nancing can be accessed for infrastructure PPPs? There clearly are factors that make a
difference in terms of financing costs involving the particulars of a project like coun-
try, sector, type of contract, risk mitigation features of the contact, and the mix of
financiers. The stronger a country’s sovereign credit ratings, the easier it seems to be
for infrastructure PPPs to raise affordable private finance (IMF 2006). In terms of sec-
tors, power generation is probably the easiest in which to raise private finance, largely
because the supply of power is relatively commercial, with cost-reflective end-user
Table 2. Breakdown of PPI financing by region, 2015, in $ billions
Region Total Investment Public Money Donors/MDBs Private Sector
Latin America 17.3 39% 14% 46%

East Asia 10.1 13% 4% 83%
SSA 6.2 13% 21% 64%

South Asia 5.4 30% 21% 49%
All Regions 39.0 27% 14% 59%
Source: Chao and Saha 2016.
tariffs being less controversial for power than for services like water and sanitation.
Electricity generation PPPs can also be purely private operations, often structured as
Independent Power Producer (IPP) projects involving ownership of the assets by the
private sector rather than by government as in traditional concessions. Private financ-
ing for power generation can also be relatively affordable because generation is not a
natural monopoly service in the same way that some other infrastructure services
are, such as water supply—multiple facilities can be built to feed power into national
grids, so generation can be relatively competitive. This puts some downward pressure
on financing costs.
IPPs may be popular, but affordable private finance for all kinds of large PPPs usu-
ally requires a host of risk mitigation mechanisms. Most IPP power purchase agree-
ments (PPAs) must be backed by security arrangements such as escrow accounts,
letters of credit, targeted subsidies, budget commitments, etc. In countries without
domestic capital markets that can finance PPP projects, PPAs often must be denomi-
nated in hard currencies such as U.S. Dollars or Euros, indexed to currency baskets, or
backed by foreign exchange liquidity facilities. Sovereign guarantees are also usually
required to back various aspects critical to PPP project cash flows and profitability,
including off-take commitments, fuel supply availability, currency convertibility and
transferability, interest rates, exchange rates, tariff rates, and revenue levels. Donors,
MDBs, as well as private institutions also provide guarantees or insurance products
to cover risks that private lenders or investors are unable or unwilling to take. Finally,
governments, donors, and MDBs have all increased their financing shares in large
PPPs to reduce the size of private project finance and help make it affordable.
But when governments assume or share project investment risks, they need to
manage conflicts of interest. For example, can governments act simultaneously as
financiers interested in the financial sustainability of projects, as well as regulators
charged with protecting the interests of end users? Will a government allow a con-
cession company in which it has invested substantial amounts of capital to declare
bankruptcy?
Leigland 117
Does it ultimately matter if private participation leads to operational efficiency, but
not more investment? Lastly, if services become less costly and their availability in-
creases, why should a lack of investment be a concern?
The problem is that PPPs tend to involve complex contracts that are difficult and
costly to prepare. Most of the time this effort and expense is justified on the basis that

more investment will be forthcoming. The absence of such investment substantiates
Klein’s point that very little separates PPPs from well-run public companies, and ar-
gues for more attention to basic management improvements and better regulation
before considering private participation.
Poverty Impacts
If a PPP generates investment, or even if it achieves only efficiency gains, it can make
possible a host of other benefits like improved service quality, affordable tariffs, or
expanded service access. In poor countries, these kinds of benefits are usually cited
as justification for involving the private sector in infrastructure service delivery. But
Estache and Philippe (2012) note that there is relatively little empirical evidence that
focuses on the development impacts of small or large-scale PPPs. What evidence there
is suggests that such projects do not provide the poor with enough affordable access
to services.
Gassner’s study, cited above, shows why this topic can generate so much contro-
versy. She compared average annual values for indicators measuring performance
before and after private sector involvement in water and electricity utilities. Private
involvement (via PPPs or privatization) generated a number of impressive efficiency
gains, but they were not associated with an equitable distribution of project bene-
fits. For example, the labor productivity gains were associated with reductions in staff
numbers for both water and electricity. Employment fell by 24 percent in electricity
and by 22 percent in water following the introduction of private participation. Utili-
ties operated by governments used considerably more employees than privately oper-
ated utilities to achieve the same level of service. Moreover, the efficiency gains were
not associated with changes in investment or average residential tariffs. Because the
gains would normally translate into lower costs for the operator, Gassner speculated
about reasons why there was no sign of the lower costs translating into greater in-
vestment or lower prices. One possible explanation was that services were already so
underpriced that even huge efficiency gains could not justify price reductions. An-
other possibility was that the private operators took advantage of weak regulatory
oversight to simply retain all of the gains as profits, passing on nothing to customers
or to cover O&M costs. This latter possibility raises questions about the long-term sus-
tainability of PPP efficiency gains.
In their survey of PPP research, Estache and Philippe (2012) concluded that in-
vestment and efficiency gains in the telecommunications sector (resulting mostly
from technological advances) have generally contributed to the wide distribution of
benefits like increased access, improved affordability and better service quality. But in
most other sectors, even when benefits result from improved efficiency or investment,
they are not always shared with customers or governments. This is because regu-
lation is not typically designed to pass on such gains to other stakeholders like res-

idential users: “For the most successful projects, unless regulation works, efficiency
gains become rents which fuel conflicts between government, users and operators,”
(Estache and Philippe 2012).
How can PPPs be done in such a way that the likelihood of developmental ben-
efits is strongly increased? PPPs that successfully generate benefits for the poor res-
idents of developing countries usually begin with the realization that such benefits
do not automatically result from increased private investment or improved efficiency.
And as long as private partners are mostly responsible for developing work plans, as
part of the bid process and later during early stages of implementation, there is no
reason to expect effective measures for benefit distribution. This is not a particularly
new problem. PPP contract designs and tariff structures that fail to benefit—or even
disadvantage—major constituencies of stakeholders have led to renegotiations and
project collapses from the early days of infrastructure concessions in the early 1990s.
Many recommendations have been made about the need to undertake stakeholder
analysis by modeling the distribution of project benefits before a contract has been
awarded (e.g., van den Berg 2000). But again, this kind of preparation requires fi-
nancial resources and expertise that must be brought into play before PPP contracts
are signed.
Governance Issues
Governance refers to the various ways in which governments’ institutional capacity
or lack thereof, intentionally or unintentionally affects the performance of PPPs. Po-
litical leadership is a rare but highly important way in which governments can facili-
tate significant infrastructure investment via PPI arrangements (Jones, Jammal, and
Gokgur 2002; Eberhard, Kolker, and Leigland 2014). But generally, strong political
support for PPPs in low-income countries is rare. A country’s investment climate can
also have a major positive impact on the development of PPP projects (IMF 2006),
but almost no low-income countries have investment grade credit ratings that would
reflect strong investment climates.
Engel, Fischer, and Galetovic (2014) argued that the governance structure under
which most PPPs operate is “usually defective.” This is largely because PPPs share
many of the same defects as standard public works, including improper design, poor
procurement, and frequent renegotiations that Engel and his co-authors conclude are
often opportunistic attempts by private contractors to increase profits. A recent study
by the World Bank’s Public Private Infrastructure Advisory Facility (PPIAF) found
Leigland 119
that in the interests of getting projects to completion, many developing countries use
limited competition and direct negotiation to procure private sponsors and operators
(PPIAF 2014). But PPIAF also found that projects procured in this manner tend to be
more expensive and subject to more problems in implementation.
Eberhard and Gratwick (2010) found that few African countries have established

the key legal and regulatory elements needed to guide private participation in their
power sectors—a clear policy statement and supporting legislation. Some countries
have passed laws permitting independent power producer (IPP) projects, but al-
most none have addressed the relationship between IPPs and state-owned energy
providers. In a more recent study, Eberhard et al. (2016) conclude that in general,
Africa’s power sector continues to reflect a partial approach to reforms that facilitate
more private investment, with little achieved over the last 20 years in terms of un-
bundling, privatization, or the introduction of competition.
After “PPP units” became prominent in countries like the United Kingdom and
Australia, they have become the preferred institutional tool to regulate the process of
PPP project development in many developing countries. A variety of international or-
ganizations have recommended establishing PPP units, including the OECD (2010),
the UN (2007), and APEC (2008). The rapid growth in the number of such units
seems to have resulted from a widespread perception that well-functioning PPP units
are important for the success of a country’s PPP program. However, a recent liter-
ature review by Lemma (2013) concluded that “there is very little quantitative evi-
dence of the value of centralized PPP coordination units...”. There seems to be con-
sensus that a large and growing number of such units exist, but they differ so widely
by country and sector that it is difficult to generalize about their value, or even agree
on how to measure their worth (Burger and Hawkesworth 2011; EPEC 2014).
The contribution of PPP units to achieving success may not be entirely clear, but
the reasons for their ineffectiveness are more so. One of the earliest studies of these
units, sponsored by PPIAF (2007), noted that they were often established due to frus-
tration on the parts of governments with the inability of existing institutions to move
PPP programs forward. But the study found that in such situations the governments
often overlooked the fact that the very shortcomings that made the existing institu-
tions underperform may well undermine the new PPP units as well (PPIAF 2007).
The PPIAF study concluded that in countries where top politicians do not support the
PPP program, where procurement of infrastructure and capital works is not transpar-
ent and competitive, and where the machinery of government is chronically uncoor-
dinated, PPP units are unlikely to be effective. The study concluded that there is little
chance that PPP units can work well where such problems persist.
One problem with PPP units that seems to have limited their usefulness is their
disappointing performance in accelerating PPP deal flow. Government officials in de-
veloping countries often expect such units to speed up the process of getting PPPs
to completion. But such units often produce exactly the opposite result—the flow of
deals slows down because of the perceived need to carefully apply standardized pro-
cedures in project development. Better-prepared projects should be more sustainable,
and that should off-set lengthy preparation periods to some degree. But this logic
rarely satisfies government officials interested in getting projects done as quickly as
possible.

South Africa’s PPP Unit, which was established in mid-2000 in the National Trea-
sury, and which boasted procedures and guidebooks that quickly became good prac-
tice examples used by other national agencies in Africa, is a case in point (Lemma
2013). The guidelines called for over two dozen relatively elaborate preparation steps,
as well as a series of National Treasury approvals of the process recommended by the
Treasury’s PPP Unit. The entire process (including the use of “public sector compara-
tors”) was quickly judged by the private sector to be too lengthy, too complex, very ex-
pensive, highly over-regulated, and unfortunately responsible for slowing down PPP
deal flow (Castalia Strategic Advisors 2007). The PPP Unit itself admitted that its re-
quired VFM studies at the municipal level averaged about 30 months (Levinsohn and
Reardon 2007). As a result, under this regulatory regime, PPP deal flow remained
stagnant in South Africa.
In 2012, the PPP Unit lost its status as a separate department within National
Treasury when it was absorbed by the Government Technical Advisory Centre
(GTAC), and its role changed from process regulator to the provider of PPP trans-
action advisory services (GTAC 2015). The experience of South Africa’s PPP Unit
seems consistent with the findings of several researchers, confirming that compli-
cated frameworks for regulating the development of PPP projects can significantly
slow the process of getting projects to financial close (Zhang and Kumaraswamy
2001; Chan et al. 2010).
PPIAF’s long experience with the creation of PPP units, particularly in Africa, sug-
gests two conditions under which these entities can add value. First, if units have no
authority to require compliance with their recommendations, they tend to be quickly
relegated to minor advisory roles without much influence. This is because in poor
countries governments cannot afford to employ the sort of experts whose advice is
of significant value to public or private sector project proponents. Sometimes experts
can be seconded to PPP units from private sector firms, or recruited for limited periods
with the help of donor funding. But these kinds of experts typically cannot be retained
long enough to transfer significant levels of professional skills to regular staff (and if
they do, regular staff members tend to leave for higher-paying jobs). To enforce gov-
ernment rules regarding how PPPs should be developed and structured, as opposed
to just providing advice, units can function reasonably well with a small central core
of expertise. The rest of the staff component can be made up of competent project
managers with more general kinds of professional skills.
Second, the rules that PPP units enforce should minimize direct pre-approval of
actions by senior government officials based on vague decision-making guidance,
Leigland 121
in favor of routine, post-audit reviews of actions based on clear, specific procedural
guidelines or regulations. Senior officials can become involved after the fact, if the
post-audit reviews reveal that proper procedures were not followed. The pre-approval
approach encourages arbitrary decision making and can slow implementation to a
standstill. But the post-audit approach is not easy to implement: it requires clear

and complete procedural guidelines or regulations, along with consistent, reasonably
high-quality auditing skills and procedures. PPP units will not work well in countries
already plagued with severe governance defects affecting standard public works pro-
curement and contracting. This is one of the reasons why PPPs are not ideal tools for
every developing country.
Transformational PPPs
In light of the preceding discussion, what is to be made of the G20’s enthusiasm
for regional or “transformational” PPP projects? Indeed, why would such projects be
any more successful than smaller and simpler PPP projects in meeting expectations
regarding costs, profits, finance, investment, equity, efficiency, and governance? Al-
though the term is widely used, there is no precise, universally-accepted definition of
“transformation project”, but as noted previously, the term is normally used to refer
to large, regional, or cross-border infrastructure projects involving private participa-
tion. Various prioritized lists of such projects have been compiled by several interlocu-
tors, including the World Bank (2011), the G20 MDB Working Group (2011) and the
African Union Commission (2012).
Such regional projects are extremely rare. SSA’s actual experience with regional
infrastructure PPI projects is almost non-existent. The World Bank’s PPI Database
has recorded only seven regional infrastructure projects on the African continent
since 2000 (all in SSA). Five have been transport projects, while; two have been nat-
ural gas transmission projects (World Bank, PPI Project Database). But it would be
misleading to imply that large regional projects are easier to do in other regions. Since
1992, the PPI Project Database has recorded only 15 cross-border projects globally,
in any infrastructure sector. The disadvantage of so little experience with large re-
gional projects is clear. Koppenjan (2008) notes that the learning curve associated
with structuring mega-projects as PPPs is extremely steep and such projects seem to
be most successful in countries and sectors with a considerable amount of PPP expe-
rience at a variety of levels.
Those who defend these large PPP projects point to the advantages of economies-
of-scale and of scope. This “bigger is better” approach is also consistent with popu-
lar theories of economic development, which argue that big development challenges
like poverty alleviation, energy scarcity, and urbanization can only be solved by “big
push” solutions involving large projects and large amounts of aid (Sachs 2006). How-
ever, these “transformational” projects frequently possess the size and complexity that
would make them subject to what Flyvbjerg would call the “iron law of megapro-
jects.” According to Flyvbjerg (2014), 90 percent of projects costing over one billion
dollars, end up “over budget, over time, over and over again.” This is a problem for
all large projects, not just PPPs, but Flyvbjerg points out that “Private capital is no
panacea for the ills in megaproject management, to be sure; in some cases, private

capital may even make things worse...” Flyvbjerg (2014). Part of the reason for this
is that the complexity of large PPPs—with multiple players, complicated and lengthy
contract negotiations, high transaction costs, forecasting uncertainties, etc.—leads
to what Ansar et al. 2016 call a low “investment fragility threshold”—a high vul-
nerability of the investments to become unrecoverable due to the impact of random
events.
With large, transformational projects, economies-of-scale are cited as a factor that
should help reduce the amount of preparation costs. But using the cost metrics dis-
cussed earlier, and even assuming some economies-of-scale, the costs of preparing
transformational PPP projects are staggering. The average total cost of the 14 re-
gional power sector projects in the Priority Action Plan of the Program for Infrastruc-
ture Development for Africa (PIDA) is $2.9 billion (AUC 2012). Preparing projects
of that size could require as much as $430 million in total development costs, with
roughly one-third of that ($140 million) being the responsibility of the government
and its development partners. These development costs tend to be more expensive
than costs incurred for developing conventional publicly procured projects, because
in addition to more complicated contracts and contact negotiations, they involve
much more up-stream preparation. Many low-income countries do not have a specific
PPP legislative framework conducive to large PPPs or a single set of procedures for
the award of a suitable PPP contract. Without such a legal-regulatory framework in
place, project sponsors need to obtain multiple licenses, permits, and authorizations
from various governmental and local authorities in order to implement the project.
All of this increases the political risk for private investors, in case one or more key
permissions cannot be obtained, and makes it more difficult to have these licenses,
permits, and authorizations assigned to the project financiers as a form of security
under the financing arrangements.
What role would private finance play in these large, regional PPPs? Inga III, a re-
gional power project which is probably on more lists of “transformational” infrastruc-
ture projects than any other in SSA, suggests some answers to this question. When
it came under serious consideration in the early 2000s, Inga III was intended to be
the third and largest hydropower project in the Inga basin of the Congo River in the
Democratic Republic of the Congo (DRC). Private sector participation has always been
described as a requirement for the project rather than an option because the com-
bination of the government plus donors and MDBs did not have the kind of capital
required for project investment (World Bank 2014). But as the project developed, it
Leigland 123
became clear that private partners would not necessarily be responsible for all or even
most of the finance required for the project.
In a “preliminary financing feasibility study” commissioned by the World Bank
(BNP Paribas 2009), the consultants estimated the total cost of the project at about
$9 billion for a facility that would generate 4,500 MW. The debt share of total costs

($6.7 billion) would be provided by four MDBs, six DFIs, several export credit agen-
cies, and Chinese policy banks. Commercial banks were expected to provide no more
than about 10 percent of the total, assuming that MDB credit enhancements could
be accessed from the World Bank’s Partial Risk Guarantee (PRG) facility and the Mul-
tilateral Investment Guarantee Agency (MIGA). The study assumed that one-third
of the equity needed for the project (about $2.2 billion) would come from the gov-
ernment, sourced as loans from MDBs. This government equity contribution (about
$730 million) would be in addition to project preparation costs that could be as much
as $400 million for a project of this size, using the metrics discussed above.
At about $1 billion, the government contribution needed to make Inga III viable as
a PPP seems untenable. In mid-2010, the government reached the completion point
under the Enhanced Heavily Indebted Poor Countries (HIPC) initiative after benefit-
ting from assistance under the Multilateral Debt Relief Initiative (MDRI). By 2013, the
DRC’s total outstanding central government debt owed to bi- and multilateral credi-
tors was $2.9 billion (IMF 2015). This total reflected a marked improvement over the
2009 level. But the DRC’s debt sustainability rating was still assessed as “moderate”
by the IMF, meaning that any sharp increase in external debt, even to MDBs or donors,
risked the return of debt distress. The wisdom of a precipitous 30 percent increase in
such debt to support Inga III would no doubt be questioned. The conclusions regard-
ing Inga III seem clear: it would not be predominantly financed by the private sector,
leaving the government with a financing burden that it could not afford. The alter-
native to a transformational PPP in this case is not a project done using traditional
public procurement. Rather, it is a collection of smaller projects (public or private)
that are more easily financed and managed.
Is there any evidence to suggest that benefits resulting from efficiencies or invest-
ment generated by large transformational PPPs would be distributed equitably across
the country’s population? In terms of equity arguments, MDBs have not been consis-
tent in their messaging. On the one hand, the World Bank (2011) argued in its infras-
tructure strategy update that “Large infrastructure projects have often been success-
ful in making project affected people the beneficiaries of the project displacing them,
as well as achieving development objectives, like the benefit sharing arrangements in
hydropower.”
But the same document (World Bank 2011) acknowledged that the Bank’s strat-
egy of pursuing economic growth through such infrastructure projects, with the ex-
pectation that benefits will “trickle down” to the poor, had not been borne out by the
facts, and that the results of this approach have often been “slow”. Poverty impacts
have proven “complex to achieve and demonstrate,” and “Learning from past expe-
riences, the Group will do more to enhance the delivery of infrastructure services to
the poor,” (World Bank 2011).
At least in the power sector, views on the socio-economic value of large regional
projects may be changing. The International Energy Agency (IEA) has acknowledged

that 70 percent of rural areas in developing countries are best electrified with mini-
grid and other off-grid solutions (mostly wind and solar), and that these kinds of solu-
tions would need to receive most of the new investments recommended for the power
sector (International Energy Agency 2011). The value of this small-scale approach
seems to be supported even in the DRC, where the government’s Inga III project direc-
tor recently acknowledged that the project will not generate many benefits for rural
areas, and that off-grid solutions involving wind, solar, and small hydropower will be
needed to adequately serve populations in those areas (Worldfolio 2016).
The larger the PPP, the more important it is to have knowledgeable and well-
capacitated partners, including governments who must play a principal role in project
preparation, procurement, and post-closure performance monitoring. In the poorest
countries, that kind of capacity does not exist, so donors and MDBs try to create it,
sometimes specifically for transformational projects.
But how effective is this kind of just-in-time institutional development? Again, Inga
III serves as an instructive example. As a condition of receiving an IDA grant for
project preparation studies in 2014, the DRC Government was asked by the World
Bank to adopt a key element of improved project governance. By 2015, the govern-
ment agreed to create a ring-fenced development authority (Agence pour le Développe-
ment et la Promotion d’Inga – ADEPI) to manage Inga’s development, and mobilize pri-
vate participation and public financing. ADEPI was supposed to be created by law
as an autonomous entity reporting to the Prime Minister’s office. It was to have a
Board of Directors representing various Inga development stakeholders. All ADEPI
staff members, including the Director, were supposed to be recruited competitively.
In other words, ADEPI was expected to act as a kind of independent PPP unit specifi-
cally for the Inga project, combining financial resources and expertise to ensure that
responsible best practice was followed in the development of the project.
The grant was made, but the government evidently did not feel compelled to meet
these obligations. On September 12, 2016, the World Bank (2016b) cancelled its
grant support to Inga III. The activity had already been suspended in July due to the
government’s decision to “take the project in a different strategic direction” to that
agreed with the Bank in 2014. At the time of suspension, only 6 percent of total
preparation financing had been disbursed; most of the studies had not been com-
pleted, and many had not been started. ADEPI had not been established as an au-
tonomous entity (World Bank 2016a). In late 2015, President Kabila moved ADEPI
into his cabinet and took personal control of the project. But this meant that the in-
dependent PPP unit functions would be lost and some of the key environmental and
Leigland 125
social studies probably would not be completed. The lesson here also seems clear: gov-
ernments without a recent history of legal, regulatory, and institutional reforms are
unlikely to make rapid, significant progress in these areas during the course of a single
large project.
Are there ways of increasing the chances that transformational PPPs will be suc-

cessful? There are very few examples of successful PPPs of transformational scale, but
Ansar and his colleagues (Ansar et al. 2016) make recommendations regarding ways
of avoiding problems with mega-projects, which apply equally well to large PPPs: (i)
be skeptical of the numbers presented at appraisal; (ii) seek to de-bias estimates of
time to task completion, costs, and benefits by demanding more extreme stress tests;
and (iii) look for evidence of meaningful cost-benefit analysis as part of appraisal.
A second way to improve the chances of mega-PPP success is to involve multiple
countries in the project and ensure that one or more of the project partners has some
PPP capacity and can take a leadership role in the project. The N4 Toll Road Con-
cession, awarded in 1997, linked South Africa’s most industrialized, but effectively
land-locked northern and eastern regions (Gauteng and Mpumalanga provinces) to
the Mozambican port of Maputo. When the project reached closure, it was recog-
nized as a pioneering accomplishment—it was the first toll road concession signed
in SSA. It was the first cross-border transport PPP, and only the second regional PPP
in any sector. The project was successful because of high-level support that came di-
rectly from the presidents of the two countries, but also because of South Africa’s
technical and managerial expertise, and their already considerable experience with
toll roads. South Africa led the project and built capacity among Mozambican coun-
terparts along the way (Thomas 2009).
A third way to avoid problems with transformational PPPs is to do smaller PPPs
instead. This is particularly the case when transformational PPPs involve massive
capital investments based on razor-thin profit margins. When possible, projects of
this kind need to be broken down into smaller projects. In the case of projects like
Inga III, this suggestion is more practical than it may sound. Since the last wave of
work on Inga III began in the late 2000s, a change in thinking about investment in
renewable power generation has occurred in Africa, thanks to South Africa’s Renew-
able Energy IPP Program (REIPPP) and the Scaling Solar Renewable Energy IPP Pro-
gram in Zambia. Over four rounds of bidding between 2012 and 2015, South Africa’s
program attracted $19 billion in private investment in 92 on-grid IPPs, totaling
6,327 MW, with wind and solar PV prices among the lowest in the world (Eberhard,
Kolker, and Leigland 2014). Instead of being primarily about climate change mitiga-
tion, or off-grid service in rural areas, REIPPP has shown how quickly renewable en-
ergy can add generating capacity to power grids, as compared with large hydropower
or coal-fired plants. Construction times for these solar projects are typically under 12
months per facility. In other words, by 2018, South Africa will have 32 percent more
generating capacity in place than Inga III is expected to provide after ten years of
construction is finished.4 The Scaling Solar program, designed by the IFC for the Zam-
bian Government, is a much smaller tender program, but has reduced power prices
even more than in South Africa (IEA 2017). REIPPP and Scaling Solar both involve
extensive pre-bid project design, government and third-party guarantees, donor and
MDB financing (of preparation as well as capital investment), and non-negotiable

project documentation.
Conclusions: The Emergence of Hybrid PPPs

PPPs have not, in general, met expectations. Based on the findings of the survey by
Burger and Hawkesworth (2011), it seems realistic to assume that PPPs may eventu-
ally account for 10 to 15 percent of public sector infrastructure investment in OECD
countries and up to 25 percent in developing countries. These are of course not in-
significant percentages, but they are nothing like what was expected of PPPs in the
1990s. In particular, the evidence shows that PPPs have never successfully gained a
foothold in the poorest countries.
The studies reviewed in this paper suggest that governments and their development
partners have not done a very effective job of anticipating PPP costs and profits, so
assessments of value for money involving tools like public sector comparators have
been frequently off the mark. This is true of all kinds of infrastructure projects, in
developed as well as developing countries, but especially so in the case of PPPs. In the
developing world, particularly in the poorest countries, PPP project costs have been
higher than expected, and profits lower. This of course has contributed to the limited
use of PPPs in these countries.
PPP project preparation costs are much higher than were expected in the 1990s,
and higher than the costs of preparing projects for traditional public procurement.
This is partly because of the upstream legal and regulatory preparation required in
many developing countries, as well as the complexity and higher costs of PPP pro-
curement compared with traditional government construction contracting (value-
for-money assessments involving tools like public sector comparators also tend to be
more rigorously applied to projects involving private participation, but most experts
argue that such tools should be used for both PPP procurement and traditional gov-
ernment contracting). High preparation costs present governments and their devel-
opment partners with a difficult problem: who pays for preparation costs that cannot
always be recovered in some way from the project?
New data compiled by the World Bank in 2016 (see table 2), suggests another sur-
prising conclusion about infrastructure PPPs: in many situations, private finance is
not the predominant source of investment funding for these projects. Governments,
or their state-owned banks, along with donors and MDBs, account for more than half
of this funding in many cases, even in middle-income regions like Latin America. This
Leigland 127
kind of public funding seems essential to attract private finance for many kinds of
projects. But this reality challenges some of the traditional definitions of PPPs, espe-
cially with regard to the role of the private partner.
Perhaps one of the least surprising findings about infrastructure PPPs is the fact
that developmental benefits in terms of poverty reduction are far from automatic.

This has been a special focus of former World Bank economist, Antonio Estache, and
his critical assessments have found their way into some of the most powerful civil so-
ciety critiques of PPPs. Estache and others point out that positive poverty impacts
must be purposely designed into projects, and then monitored by regulators (Estache
and Rossi 2004; Andrés, Schwartz, and Guasch 2013). Private partners do not typ-
ically pursue these benefits without substantial incentivization by governments and
donors.
Governance problems can have significant negative impacts on PPP project devel-
opment and oversight. Governments must play a strong role in PPPs, but often the
ultimate justification for doing a project as a PPP is the weakness of host government
institutions and their lack of skilled staff. PPP units can help, but cannot be expected
to work more effectively than other existing governmental agencies. It is not always
an exaggeration to say that a government capable of fully playing its role in design-
ing, developing, implementing, and regulating PPPs is probably better off using tradi-
tional public procurement to achieve the same objectives. This supports Klein’s point
that well-run public companies match the performance of private firms in regulated
sectors.
The bigger the PPP, the more prominent are all of these problems—“bigger is bet-
ter” does not seem to apply to these kinds of projects. Part of the problem is simply
the size of the project, but doing a project as a PPP accentuates the problems cre-
ated by size. This is the reason why transformational PPPs are so rare, and existing
projects like Inga III have encountered so many problems. But the best alternative to a
transformational PPP is not necessarily a project done via traditional public procure-
ment. It is scaling them down to smaller projects, or a collection of smaller projects,
done as either public or private projects. Smaller projects reduce the overall invest-
ment fragility of these undertakings, making them more easily financeable. What
still seems unclear is why the G20 and other international groups continue to pro-
mote mega-PPPs as solutions to infrastructure problems in poor countries. Civil soci-
ety groups seem to have taken the lead in criticizing the G20’s approach (Alexander
2013). But despite the fact that these groups now routinely cite studies making up
the emerging evidence-based critique of PPPs, their opinions have had no noticeable
impact on G20 discussions.
If PPPs continue to account for 15 to 25 percent of infrastructure investment,
this suggests that governments and their development partners may be finding
some situations in which conditions are right for PPPs to provide value for money.
The emerging evidence-based critique of PPPs suggests the type of conditions
required, but these are not the same ones envisioned in the 1990s. It is no longer
assumed that self-sustaining PPPs that fully cover all operating and capital costs
are workable and desirable in a large variety of infrastructure service delivery situa-
tions in poor countries. The World Bank long ago stopped advising governments that
were granting concessions to insist on full cost-recovery tariffs, as it did in the 1990s

(Kerf et al. 1998).
The evidence-based critique of PPPs suggests that compared with the traditional
PPP model promoted in the 1990s, successful PPPs today tend to be hybrids, com-
bining some elements of traditional PPPs with those of consulting or EPC contracts.
These hybrids involve less of a role, and lower risks, for private sector operators and fi-
nanciers, and more of a role (and more risks) for governments and their development
partners. This shift in roles is evident across the spectrum of criticisms discussed in
this paper: (i) “blended” or subsidized finance now plays a much stronger role in en-
suring private sector profits from PPPs; (ii) project preparation is no longer left solely
to private partners, but is increasingly paid for and managed by governments and
donors; (iii) significant amounts of private finance are available only if governments
and MDBs provide guarantees and other risk-mitigation mechanisms and share the
financing burden using public or concessional money; (iv) developmental benefits
are achieved when projects are designed to include them, and this design work is
no longer left to the private sector; and (v) governments must play a strong role in
the PPP project development and oversight, but improving governance in poor coun-
tries typically requires considerable amounts of funding and pressure from donors
and MDBs. In situations where even hybrid projects are difficult to carry out, mainly
in the poorest countries or with mega “transformational” scale projects, PPPs have
not been very successful, and are no longer frequently attempted.
Notes
1. This database defines “PPI projects” to include management contracts, utility privatizations, and
merchant projects, as well as projects more widely thought of as PPPs.
2. In a 1998 publication that asked authors to speculate on the future of MDBs like the World Bank
and EIB, Klein “looked back” from a 2044 vantage point and noted that “...the private provision of most
economic and social services rendered the funding and guarantee functions of the World Bank group
largely superfluous,” (Klein, 1998).
3. Examples are Africa50, established by the African Development Bank, and the World Bank
Group’s Global Infrastructure Facility (GIF), both of which emphasize project preparation as well as fi-
nancing.
4. Wind and solar power are much more variable than hydropower, and cannot provide the base
load capacity that most developing countries need. Thus, this comparison between large hydropower
and wind/solar is somewhat unfair. But when combined with small hydropower plants or gas-based
generation, wind and solar could dramatically reduce the need for a massive hydropower investment
in a country like DRC, especially if the primary objective is increasing energy access for the largely rural
population.
Leigland 129
References
African Development Bank. 2013. “Africa50: Questions and Answers.” Memorandum to the AfDB Board
of Directors from Secretary General, C. Akintomide, July 16.
African Union Commission. 2012. Program for Infrastructure Development in Africa: Interconnecting,

integrating and transforming a continent. Africa Union Commission (AUC), African Development
Bank (AfDB), UN Economic Commission for Africa (UNECA), NEPAD Planning and Coordinat-
ing Agency (NPCA) (January). Available at: https://www.afdb.org/en/topics-and-sectors/initiatives-
partnerships/programme-for-infrastructure-development-in-africa-pida/.
Alcazar, L., M. A. Abdala, and M. M. Shirley. 2000. “The Buenos Aires Water Concession.” Policy Re-
search Working Paper No. 2311, World Bank, Washington, DC.
Alexander, N. 2013. Responsible Investment in Infrastructure: Recommendations for the G20. Washing-
ton, DC: Heinrich Böll Foundation North America (October). Available at: https://us.boell.org/sites/
default/files/responsible_investment_in_infrastructure.pdf .
Andrés, L., J. Schwartz, and L. Guasch. 2013. Uncovering the Drivers of Utility Performance: Lessons from
Latin America and the Caribbean on the Role of the Private Sector, Regulation, and Governance in the Power,
Water, and Telecommunication Sectors. Washington, DC: World Bank.
Ansar, A., B. Flyvbjerg, A. Budzier, and D. Lunn. 2016. “Big Is Fragile: An Attempt at Theorizing Scale.”
In The Oxford Handbook of Megaproject Management, edited by Bent Flyvbjerg, Oxford: Oxford Univer-
sity Press.
Asia-Pacific Economic Cooperation. 2008. “Joint Ministerial Statement: 2008.” 15th Finance Ministers’
Meeting, Trujillo, Peru, 5-6 November.
Bain, R. 2009. “Error and Optimism Bias in Toll Road Traffic Forecasts.” Transportation 36 (5): 469–82.
Bhattacharya, A., M. Romani, and N. Stern. 2012. “Infrastructure for Development: Meeting the
Challenge.” Policy paper sponsored by The Centre for Climate Change Economics and Policy, the
Grantham Research Institute on Climate Change and the Environment, in collaboration with the In-
tergovernmental Group of Twenty-Four on International Monetary Affairs and Development (G-24)
(June). Available at: http://www.lse.ac.uk/GranthamInstitute/wp-content/uploads/2014/03/PP-
infrastructure-for-development-meeting-the-challenge.pdf .
Blanc-Brude, F., H. Goldsmith, and T. Välilä. 2006. Ex Ante Construction Costs in the European Road Sector:
A Comparison of Public-Private Partnerships and Traditional Public Procurement. Luxembourg: European
Investment Bank.
BNP Paribas. 2009. “Preliminary Financial Feasibility Study—Project Inga 3.” Report prepared for the
World Bank and PPIAF, December.
Bosshard, P. 2012. Infrastructure for Whom? A Critique of the Infrastructure Strategies of the Group of 20
and the World Bank. Berkeley, CA: International Rivers. Available at: https://www.internationalrivers.
org/sites/default/files/attached-files/infrastructure_for_whom_report.pdf .
Bretton Woods Project. 2016. “World Bank Rolls out the Carpet for “Troubled Megaprojects” and
PPPs.” April 5. Available at: http://www.brettonwoodsproject.org/2016/04/world-bank-rolls-out-
the-carpet-for-troubled-megaprojects/.
Burger, P., and I. Hawkesworth. 2011. “How to Attain Value for Money: Comparing PPP and Tradi-
tional Infrastructure Public Procurement.” OECD Journal on Budgeting 2011(1): 1–56. Available at:
https://www.oecd.org/gov/budgeting/49070709.pdf .
Castalia Strategic Advisors. 2007. “Key Challenges to Public Private Partnerships in South Africa: Sum-
mary of Interview Findings.” Report to the Support Programme for Accelerated Infrastructure Devel-
opment (SPAID), jointly sponsored by the Presidency of the Republic of South Africa and the Business
Trust, Pretoria, April.
———. 2010. “School PPPs in New Zealand: Will PPPs Provide Value for Money as a Method of
Procuring Schools in New Zealand? Working Paper No. 2: Cost Benefit Analysis.” Report to the Aus-
tralian Department of Education and Training, Canberra.
Cambridge Economic Policy Associates. 2012. Infrastructure Consortium for Africa (ICA) Assessment of
Project Preparation Facilities for Africa. Report to the ICA Secretariat, in Association with Nodalis Con-
seil, London, September.

CEE Bankwatch Network. 2008. Never Mind the Balance Sheet: The Dangers Posed by Public-Private
Partnerships in Central and Eastern Europe. Prague: CEE Bankwatch Network. Available at:
http://bankwatch.org/documents/never_mind_the_balance_sheet.pdf .
Chan, A. P. C., P. T. I. Lam, D. W. M. Chan, E. Cheung, and Y. Ke. 2010. “Critical Success Factors for
PPPs in Infrastructure Developments: Chinese Perspective.” Journal of Construction Engineering and
Management 136 (5): 484–94.
Chao, J. 2016. How are PPPs Really Financed? Washington, DC: World Bank. Available at:
http://blogs.worldbank.org/ppps/how-are-ppps-really-financed.
Chao, J., and D. Saha. 2016. Sources of Financing for PPP Investments 2015. Washington DC: World
Bank. Available at: https://ppi.worldbank.org/∼/media/GIAWB/PPI/Documents/Data-Notes/
2015-PPP-Investments-Sources.pdf .
De Schepper, S., E. Haeqendonck, and M. Dooms. 2015. “Understanding Pre-Contractual Transaction
Costs for Public-Private Partnership Infrastructure Projects.” International Journal of Project Manage-
ment 33 (4): 932–46. Available at: www.sciencedirect.com.
Dudkin, G., and T. Valila. 2005. Transaction Costs in Public-Private Partnerships: A First Look at the Evidence.
Luxembourg: EIB.
Eberhard, A., and K. Gratwick. 2010. IPPs in Sub-Saharan Africa: Determinants of Success. Management
Programme in Infrastructure Reform and Regulation. Cape Town: University of Cape Town, Graduate
School of Business.
Eberhard, A, K. Gratwick, E. Morella, and P. Antman. 2016. Independent Power Projects in Sub-Saharan
Africa: Lessons from Five Key Countries. Washington, DC: World Bank.
Eberhard, A, J. Kolker, and J. Leigland. 2014. South Africa’s Renewable Energy IPP Program: Success Factors
and Lessons. Washington, DC: World Bank. Available at: www.ppiaf.org.
Engel, E., R. Fischer, and A. Galetovic. 2014. The Economics of Public-Private Partnerships: A Basic Guide.
Cambridge: Cambridge University Press.
Estache, A., and C. Philippe. 2012. “The Impact of Private Participation in Infrastructure in Develop-
ing Countries: Taking Stock of about 20 Years of Experience.” ECARES Working Paper, 2012–043,
Universite Libre de Bruxelles, Brussels.
Estache, A, and M. Rossi. 2004. “Have Consumers Benefited from the Reforms in the Electricity Distribu-
tion Sector in Latin America?” Policy Research Working Paper No. 3420, World Bank, Washington
DC.
European PPP Expertise Centre. 2014. Establishing and Reforming PPP Units: Analysis of EPEC Member
PPP Units and Lessons Learnt. Luxembourg: European Investment Bank.
Farajian, M. 2010. “Transaction Cost Model for Infrastructure Public Private Partnerships in the U.S.”
Master’s Thesis, University of Maryland, Graduate School of Engineering, College Park.
Farquharson, E., and J. Encinas. 2010. The UK Treasury Infrastructure Finance Unit: Supporting PPP Fi-
nancing During the Global Liquidity Crisis. Washington DC: World Bank (March).
Flyvbjerg, B. 2005. “Policy and Planning for Large Infrastructure Projects: Problems, Causes, Cures.”
Policy Research Working Paper No. 3781, World Bank, Washington DC.
Flyvbjerg, B. 2014. “What You Should Know about Megaprojects and Why: An Overview.” Project Man-
agement Journal 45 (2): 6–19.
Leigland 131
Foster, V., and C. M. Briceño-Garmendia. 2010. Africa’s Infrastructure: A Time for Transformation. Wash-
ington DC: World Bank.
G20 Multilateral Development Bank, Working Group on Infrastructure. 2011. “Infrastruc-
ture Action Plan: Submission to the G20 by the MDB Working Group on Infrastructure.”
Washington, DC: World Bank. Available at: http://documents.worldbank.org/curated/en/
828751468331900533/pdf/655610BR0v10Se0Official0Use0Only090.pdf .

Gassner, K., A. Popov, and N. Pushak. 2009. Does Private Participation Improve Performance in Electricity
and Water Distribution? Trends and Policy Options No. 6. Washington DC: World Bank. Available at:
www.ppiaf.org.
Government Technical Advisory Centre. 2015. Government Technical Advisory Centre and National Trea-
sury Programme 8: Technical Support and Development Finance, Annual Report 2014/15. Pretoria: South
African National Treasury. Available at: http://www.gtac.gov.za.
Guasch, J. L. 2004. Granting and Renegotiating Infrastructure Concessions: Doing It Right. Washington DC:
World Bank.
Hall, D. 2015. Why Public-Private Partnerships Don’t Work: The Many Advantages of the Public Alterna-
tive. Greenwich, UK: Public Services International Research Unit. Available at: http://www.world-
psi.org/sites/default/files/rapport_eng_56pages_a4_lr.pdf .
Independent Evaluation Group. 2010. Cost-Benefit Analysis in World Bank Projects. Washington, DC:
World Bank (June).
———. 2014. World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client
Countries, FY02–12. Washington, DC: World Bank.
International Energy Agency. 2011. Energy for All, Special Early Excerpt of the World Energy Out-
look 2011. Paris: IEA. Available at: https://www.iea.org/publications/freepublications/publication/
weo2011_energy_for_all.pdf .
———. 2017. Renewable Energy Auctions: Analysing 2016. Paris: IEA. Available at: http://www.
irena.org/publications/2017/Jun/Renewable-Energy-Auctions-Analysing-2016.
International Finance Corporation. 2012. “Blended Finance at IFC: IFC’s Approach to Blending Conces-
sional Funds.” Report prepared for the IFC Board of Directors, IFC, Washington, DC, March 15.
International Monetary Fund. 2006. Determinants of Public-Private Partnerships in Infrastructure. IMF
Working Paper, IMF Institute. Washington, DC: IMF Institute.
———. 2015. “Democratic Republic of the Congo: Staff Report for the 2015 Article IV Consultation –
Debt Sustainability Analysis.” Washington, DC: IMF. Available at: https://www.imf.org/external/
pubs/ft/dsa/pdf/2015/dsacr15280.pdf .
Izaguirre, A. K. 2009. “Impact of the Financial Crisis on the Closure of Wew Infrastructure Projects
with Private Participation: A Preliminary Assessment as of January 2009.” Presentation to
the Finance, Economics, and Urban (FEU) Department of the World Bank, Washington, DC,
March 30.
Jomo, K. S., A. Chowdhury, K. Sharma, and D. Platz. 2016. “Public-Private Partnerships and the 2030
Agenda for Sustainable Development: Fit for Purpose?” UN Department of Economic & Social Affairs,
Working Paper No. 148ST/ESA/2016/DWP/148, United Nations, New York.
Jones, L., U. Jammal, and N. Gokgur. 2002. “Impact of Privatization in Cote D’Ivoire.” Report Prepared
by the Boston Institute for Developing Economies (BIDE) for Privatization Committee, Boston, July
15.
Kerf , M., R. D. Gray, T. Irwin, C. Levesque, and R. R. Taylor. 1998. “Concessions for Infrastructure — A
Guide to Their Design and Award.” Technical Paper No. 399, World Bank, Washington, DC.
Klein, M. 1998. “One Hundred Years after Bretton Woods: A Future History of the World Bank Group.”
In EIB Papers: International Financial Institutions in the 21st Century 3(2): 30–59.
———. 2015. “Public-Private Partnerships: Promise and Hype.” Policy Research Working Paper No. 7340,
Koppenjan, J. 2008. “Public–Private Partnerships and Mega-projects.” In Decision-Making on Mega-
Projects: Cost–Benefit Analysis, Planning and Innovation, edited by B. Flyvbjerg, H. Priemus and B. van
Wee, Northampton MA: Edward Elgar Publishing.
Kotze, R., A. Ferguson, and J. Leigland. 2000. “Government Facilitation of Public-Private Infrastructure

Projects: Lessons from South Africa.” Journal of Project Finance 6 (1): 47–64.
Lemma, A. 2013. “Literature Review: Evaluating the Costs and Benefits of Centralised PPP Units.”
Overseas Development Institute, Economic and Private Sector PEAKS, April. Available at:
https://assets.publishing.service.gov.uk/media/57a08a49e5274a31e0000510/Evaluating_the_
Costs_and_Benefits_of_Centralised_PPP_Units_24.pdf .
Levinsohn, D., and D. Reardon. 2007. “Municipal PPP Projects in South Africa: Obstacles and Opportu-
nities.” IP3 Public-Private Partnership Information Series, Washington, DC.
Lin, J. Y., and D. Doemeland. 2012. “Beyond Keynesianism: Global Infrastructure Investments in Times
of Crisis.” Policy Research Working Paper 5940, World Bank, Washington, DC.
Marin, P. 2009. “Public-Private Partnerships for Urban Water Utilities: A Review of Experiences in De-
veloping Countries.” PPIAF Trends and Policy Options Series No. 8, World Bank, Washington, DC.
Available at: www.ppiaf.org.
Marriott, A. 2014. “A Dangerous Diversion: Will the IFC‘s Flagship Health PPP Bankrupt Lesotho‘s Min-
istry of Health?” Oxfam Briefing Note, Oxfam International, Boston, MA. Available at: https://www.
oxfam.org/sites/www.oxfam.org/files/bn-dangerous-diversion-lesotho-health-ppp-070414-en.pdf .
Organisation for Economic Co-operation and Development. 2008. Public-Private Partnerships: In Pursuit
of Risk Sharing and Value for Money. Paris: OECD Publishing.
———. 2010. Dedicated Public Private Partnership Units: A Survey of Institutional and Governance Structure.
Paris: OECD Publishing.
Perelman, S., and T. Serebrisky. 2012. “Measuring the Technical Efficiency of Airports in Latin Amer-
ica.” Utilities Policy 22: 1–7 (September).
Pozzo di Borgo, P., A. Labeau, R. Eskinazi, J. Dehornoy, A. Parte, and M. Ameziane. 2006. “Sub-Saharan
Africa: Review of Selected Railway Concessions.” Africa Transport Sector Report No. 36491, World
Bank, Washington, D.C.
Public-Private Infrastructure Advisory Facility. 2007. Public-Private Partnership Units: Lessons for their
Design and Use in Infrastructure. Washington DC: World Bank. Available at: www.ppiaf.org.
———. 2014. Unsolicited Proposals – An Exception to Public Initiation of Infrastructure PPPs: An Analysis
of Global Trends and Lessons Learned. Washington, DC: World Bank. Available at: www.ppiaf.org.
Republic of Sierra Leone. 2015. Bidding Documents for a Management Contractor for Electricity Distribution
and Supply Authority. Freetown: Ministry of Energy.
Republic of South Africa. 2014. Brics and Africa: Partnership for Development, Integration and Industrial-
ization. Pretoria: Madiba Communications.
Ruiz-Nunez, F. 2016. PPI Investments in IDA Countries, 2011 to 2015. Washington, DC: World Bank.
Available at: https://ppi.worldbank.org/∼/media/GIAWB/PPI/Documents/Data-Notes/PPI-in-IDA-
Countries-2011-2015.pdf .
Sachs, J. D. 2006. The End of Poverty: Economic Possibilities for Our Time. New York: Penguin.
Shaoul, J. 2008. “Market Based Reforms in the Provision and Financing of Healthcare in the UK.” Kur-
swechel 2: 47–57.
Sirtaine, S., M. E. Pinglo, J. L. Guasch, and V. Foster. 2004. How Profitable are Infrastructure Concessions in
Latin America? Empirical Evidence and Regulatory Implications. Washington, DC: World Bank. Available
at: www.ppiaf.org.
Leigland 133
Thomas, R. 2009. “Development Corridors and Spatial Development Initiatives in Africa.” Re-
port prepared for Africa Capital Rising, Johannesburg. Available at: http://www.afriscapital.com/
images/stories/Development%20Corridors%20and%20Spatial%20Development%20Initiatives.pdf .
U.K. Audit Commission. 2003. PFI in Schools. London: U.K. Audit Commission.
U.K. National Audit Office. 2015. “The Choice of Finance for Capital Investment.” Briefing by the
National Audit Office, HM Treasury, NAO Communications, London. Available at: https://www.

nao.org.uk/wp-content/uploads/2015/03/The-choice-of-finance-for-capital-investment.pdf .
United Nations. 2007. Guidebook on Promoting Good Governance in Public-Private Partnerships. New York:
United Nations.
van den Berg, C. 2000. “Water Concessions: Who Wins, Who Loses, and What to Do about It.” Note No.
217, Private Infrastructure Notes, World Bank, Washington, DC.
Vassallo, J. M. 2007. “Why Traffic Forecasts in PPP Contracts are Often Overestimated?” EIB University
Research Sponsorship Programme. Luxembourg: EIB.
World Bank PPI Project Database, 1995 to 2015. Accessed February 12-15, 2017. Available at:
http://ppi.worldbank.org/customquery?mode=1.
World Bank. 2011. Transformation Through Infrastructure: World Bank Group Infrastructure Strategy Update
FY12-15. Washington, DC: World Bank.
———. 2014. “Project Appraisal Document on a Proposed Grant in the Amount of SDR 47.7 Mil-
lion (US$ 73.1 Million Equivalent) to the Democratic Republic of Congo for an INGA 3 Basse Chute
and Mid-Size Hydropower Development Technical Assistance Project.” Report No. 77420-ZR, World
Bank, Washington, DC.
———. 2016a. The World Bank Implementation Status & Results Report: DRC Inga 3 and Mid-Size Hy-
dropower Development TA (P131027). Washington, DC: World Bank.
———. 2016b. World Bank Group Suspends Financing to the Inga-3 Basse Chute Technical Assistance
Project. Washington, DC: World Bank. Available at: http://www.worldbank.org/en/news/press-
release/2016/07/25/world-bank-group-suspends-financing-to-the-inga-3-basse-chute-technical-
assistance-project.
World Bank Institute and Public-Private Infrastructure Advisory Facility. 2013. “Value-for-Money
Analysis—Practices and Challenges: How Governments Choose When to Use PPP to Deliver Pub-
lic Infrastructure and Services.” Report from World Bank Global Round-Table, 28 May, 2013, World
Bank, Washington DC. Available at: www.ppiaf.org.
Worldfolio. 2016. “‘Coherence & Efficiency’ Define New Energy Approach: Interview with Bruno
Kapandji Kalala, Project Director for the Grand Inga Development and Promotion Agency
(ADPI).” Worldfolio, September 20. Available at: http://www.theworldfolio.com/interviews/
coherence–efficiency-define-new-energy-approach/4178/
Zhang, X., and M. Kumaraswamy. 2001. “Procurement Protocols for Public-Private Partnered Projects.”
Journal of Construction Engineering and Management 127 (5): 351–8.
Zitron, J. 2006. “Public-Private Partnership Projects: Towards a Model of Contractor Bidding Decision-
Making.” Journal of Purchasing and Supply Management 12 (2): 53–62.
Subscriptions
A subscription to The World Bank Research Observer (ISSN 0257-3032) comprises 2 issues. Prices include postage; for subscribers
outside the Americas, issues are sent air freight.
Annual Subscription Rate (Volume 33, 2 issues, 2018)
Academic libraries
Print edition and site-wide online access: US$163/£175/€263
Print edition only: US$242/£160/€242
Site-wide online access only: US$193/£129/€193
Corporate
Print edition and site-wide online access: US$398/£265/€398
Print edition only: US$364/£243/€364
Site-wide online access only: US$294/£193/€294
Personal

Print edition and individual online access: US$72/£49/€72
Please note: US$ rate applies to US & Canada, Euros€ applies to Europe, UK£ applies to UK and Rest of World.
Readers with mailing addresses in non-OECD countries and in socialist economies in transition are eligible to receive complimentary
subscriptions on request by writing to the UK address below.
There may be other subscription rates available; for a complete listing, please visit https://academic.oup.com/wbro/subscribe.
Full pre-payment in the correct currency is required for all orders. Payment should be in US dollars for orders being delivered to the
USA or Canada; Euros for orders being delivered within Europe (excluding the UK); GBP sterling for orders being delivered elsewhere
(i.e., not being delivered to USA, Canada, or Europe). All orders should be accompanied by full payment and sent to your nearest Oxford
Journals office. Subscriptions are accepted for complete volumes only. Orders are regarded as firm, and payments are not refundable.
Our prices include Standard Air as postage outside of the UK. Claims must be notified within four months of despatch/order date
(whichever is later). Subscriptions in the EEC may be subject to European VAT. If registered, please supply details to avoid unnecessary
charges. For subscriptions that include online versions, a proportion of the subscription price may be subject to UK VAT. Subscribers
in Canada, please add GST to the prices quoted. Personal rate subscriptions are only available if payment is made by personal cheque
or credit card, delivery is to a private address, and is for personal use only
Back issues: The current year and two previous years’ issues are available from Oxford University Press. Previous volumes can
be obtained from the Periodicals Service Company, 11 Main Street, Germantown, NY 12526, USA. E-mail: psc@periodicals.com.
Tel: (518) 537-4700. Fax: (518) 537-5899.
Contact information: Journals Customer Service Department, Oxford University Press, Great Clarendon Street, Oxford OX2 6DP,
UK. E-mail: jnls.cust.serv@oup.com. Tel: +44 (0)1865 353907. Fax: +44 (0)1865 353485.
In the Americas, please contact: Journals Customer Service Department, Oxford University Press, 2001 Evans Road, Cary, NC
27513, USA. E-mail: jnlorders@oup.com. Tel: (800) 852-7323 (toll-free in USA/Canada) or (919) 677-0977. Fax: (919) 677-1714.
In Japan, please contact: Journals Customer Service Department, Oxford University Press, 4-5-10-8F Shiba, Minato-ku, Tokyo,
108-8386, Japan. E-mail: custserv.jp@oup.com. Tel: +81 3 5444 5858. Fax: +81 3 3454 2929.
Postal information: The World Bank Research Observer (ISSN 0257-3032) is published twice a year, in February and August, by
Oxford University Press for the International Bank for Reconstruction and Development/THE WORLD BANK. Postmaster: send address
changes to The World Bank Research Observer, Journals Customer Service Department, Oxford University Press, 2001 Evans Road, Cary,
NC 27513-2009. Periodicals postage paid at Cary, NC and at additional mailing offices. Communications regarding original articles
and editorial management should be addressed to The Editor, The World Bank Research Observer, The World Bank, 1818H Street, NW,
Washington, D.C. 20433, USA.
Environmental and ethical policies: Oxford Journals, a division of Oxford University Press, is committed to working with
the global community to bring the highest quality research to the widest possible audience. Oxford Journals will protect
the environment by implementing environmentally friendly policies and practices wherever possible. Please see academic.oup.
com/journals/pages/authors_ethics for further information on environmental and ethical policies.
Digital Object Identifiers: For information on dois and to resolve them, please visit www.doi.org.
Permissions: For information on how to request permissions to reproduce articles or information from this journal, please visit
academic.oup.com/journals/pages/authors.
Advertising: Advertising, inserts, and artwork enquiries should be addressed to Advertising and Special Sales, Oxford Journals,
Oxford University Press, Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44 (0)1865 354767; Fax: +44(0)1865 353774;
E-mail: jnlsadvertising@oup.com.
Disclaimer: Statements of fact and opinion in the articles in The World Bank Research Observer are those of the respective authors
and contributors and not of the International Bank for Reconstruction and Development/THE WORLD BANK or Oxford University Press.
Neither Oxford University Press nor the International Bank for Reconstruction and Development/THE WORLD BANK make any repre-
sentation, express or implied, in respect of the accuracy of the material in this journal and cannot accept any legal responsibility or
liability for any errors or omissions that may be made. The reader should make her or his own evaluation as to the appropriateness or
otherwise of any experimental technique described.
Paper used: The World Bank Research Observer is printed on acid-free paper that meets the minimum requirements of ANSI Standard
Z39.48-1984 (Permanence of Paper).
Indexing and abstracting: The World Bank Research Observer is indexed and/or abstracted by ABI/INFORM, CAB Abstracts, Current
Contents/Social and Behavioral Sciences, Journal of Economic Literature/EconLit, PAIS International, RePEc (Research in Economic Papers),
Social Services Citation Index, and Wilson Business Abstracts.
Copyright C 2018 The International Bank for Reconstruction and Development/THE WORLD BANK
All rights reserved; no part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise without prior written permission of the publisher or a license
permitting restricted copying issued in the UK by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1P 9HE,
or in the USA by the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
Typeset by APTARA, Noida, India; Printed by SCI, Secaucus, NJ.

Privatization The-World-Bank-Research-Observer-33-1 PDF

Uploaded by

Document Informationclick to expand document information

Document Informationclick to expand document information

Copyright:

Available Formats

Privatization The-World-Bank-Research-Observer-33-1 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Privatization The-World-Bank-Research-Observer-33-1 PDF

Uploaded by

Copyright:

Available Formats

Public Disclosure Authorized

Downloaded from https://academic.oup.com/wbro/issue/33/1 by Joint Bank-Fund Library user on 09 August 2019

Public Disclosure Authorized

Volume 33, Issue 1

Public Disclosure Authorized

ISSN 0257-3032 (PRINT)

WBRO_33_1_Cover.indd 1 20/03/18 9:13 PM

The Whys of Social Exclusion: Insights from Behavioral

Downloaded from https://academic.oup.com/wbro/issue/33/1 by Joint Bank-Fund Library user on 09 August 2019

Generalization in the Tropics – Development Policy, Randomized

Privatization in Developing Countries: What Are the Lessons of

Public-Private Partnerships in Developing Countries: The

The “Schematizing Power of Institutions”2

Hoff and Walsh 3

Barrier 1: Implicit discrimination

Salience is a central theme of behavioral economics. If individuals were unbound-

Hoff and Walsh 7

Identification of the impact of implicit discrimination leads immediately to the ques-

Hoff and Walsh 9

Hoff and Walsh 11

Hoff and Walsh 13

Interventions to Weaken the Influence of Negative Stereotypes on Self-assessment

Hoff and Walsh 15

Barrier 3: The Challenge of Adapting to Two Worlds

Hoff and Walsh 17

Particularly for males who live in dangerous neighborhoods, there is generally no

Hoff and Walsh 19

Hoff and Walsh 21

Barrier 4: “Adaptive Preferences”

Hoff and Walsh 23

What can change dysfunctional “adaptive preferences”? By observing others, indi-

a change in their perception of the costs—psychic and otherwise—of reporting the

Hoff and Walsh 27

Hoff and Walsh 29

Hoff and Walsh 31

Hoff and Walsh 33

Jörg Peters, Jörg Langbein, and Gareth Roberts

Theoretical Background and Definition of External Validity

Understanding what external validity exactly is and how it might be threatened is

Potential Hazards to External Validity

Methods and Data

Hawthorne and John Henry Effect: Does the paper

Specific Sample Problems: Does the paper

Special Care: Does the paper

Appendix A: Reviewed Papers and Ratings

Aker et al. No N/A Yes Yes Yes Yes No NGO Yes

Burde and No N/A No No Yes No No NGO No

Jensen (2010) No N/A No Yes Yes Yes No Researcher Yes

Oster and No N/A No No Yes Yes No Researcher Yes

Author Reason for exclusion

Adhvaryu (2014) Quasi-experiment

Downloaded from https://academic.oup.com/wbro/article-abstract/33/1/65/4951686 by Joint Bank-Fund Library user on 08 August 2019

Saul Estrin and Adeline Pelletier

Downloaded from https://academic.oup.com/wbro/article-abstract/33/1/65/4951686 by Joint Bank-Fund Library user on 08 August 2019

Privatization Trends: Stylized Facts

Downloaded from https://academic.oup.com/wbro/article-abstract/33/1/65/4951686 by Joint Bank-Fund Library user on 08 August 2019

Estrin and Pelletier 67