Yorking Paper Sub-series - Zimbabwe Rural and Regional Planning - No. 1 MOTIVATIONS AND MANIPULATIONS SOME PRACTICES OF PROJECT APPRAISAL AND EVALUATION D.R. Gasper December 1987 TABLE OF CONTENTS 1. Theorists and tradesmen 1.1 The imbalance in theory and training 1.2 Limits to determinate assessment: the scope for judgement, bargaining and massage 1.3 The need for some sociology of assessment 1.4 Coverage and objectives 2. Themes; doers and checkers - on the sociology of project analysis 2.1 Some roles and contrasts 2.2 "Insiders" and "Outsiders" 2.3 "Doers" 2.4 "Deliberators" 2.5 Claimers and checkers - negotiation in administrative allocation 3. Tactics; how to sell/save/slay a programme: some pathologies of project analysis 3.1 Occasions and battlegrounds 3.2 Minimization and "sinking" of costs 3.3 Framing the analysis 3.4 Generation and interpretation of data on demand and impact 3.5 Describing objectives and benefits 3.6 Synthesis of direct and indirect costs and benefits 4. Project assessment in perspective 4.1 Questions 4.2 Limited consolations 4.3 More roles than one 4.4 Perspectives for research and training Appendix: Base cases and sunk costs in rural road projects Notes to sections References -1- 1. THEORISTS AND TRADESMEN 1.1 The imbalances in theory and training This paper projects looks at aspects of offence and defence in bargaining over in their appraisal and evaluation stages. It draws on experiences with project planning as encountered both in the world of government and consultancy, and that of universities and textbooks; and tries to reduce and illuminate the gap between these two worlds, of tradesmen and theorists. the tradesmen's In real world the practice of assessment includes motives and manipulations too little considered in theorizing and training and somewhat limited resources to counter them. Behind the technicalities of assessment methods lie several sorts of bargaining. Indeed some technicalities provide avenues for bargaining. The subject of the paper is not the precise extent of such bargaining and massaging, more the types of relations and ploys involved. Identifying these types, as in Sections 2 and 3 below, is however in any case necessary for study of their extent and effects, as previewed in Section 4. Sophisticated project appraisal systems received great attention in research and training in the 1960's and 70's. (1977) noted, practical typically realities of Theorists were however, as Carruthers with - relatively speaking - little experience of planning in LDCs. Johnson (1981) too commented on the division between theorists and what one may call tradesmen. Some practical theorists have appeared though, to elucidate tradesmen's complaints, effectively suggesting that many economic theorists' allocations of their Vilson 1978). improving able, own resources had been significantly suboptimal (Bridger 1982; F. More important than great refinement in shadow pricing is the quality of project information, while keeping methods manage- and implementation paying greater issues. The attention to criticisms apply project not alternatives least to and training -2- materials, which were for a time dominated by the abstracted and tech- nocratic economic assessment of known or given alternatives. The growth of interest in ex-post evaluation is partly a response to these criticisms, aiming appraisal. It ments was showed fashionable to feed back relevant experience into design and itself a source of criticisms, since ex-post reassess- up the shadow spurious accuracy and pointless refinement of pricing when other vital questions had been neglected. But quite often evaluation work has itself been criticized in similar ways: (a) neglecting alternatives and understanding of project experience; (b) absorbed with elaborate methods; too slow and late; not relevant in orientation and expression programmes; quent (d) to potential inconsistent, users; (c) biased against exploratory inconclusive and open to manipulation; fre- either self-serving window-dressing or ignored. some This paper considers of these themes in experiences of appraisal and evaluation. It offers a post-experience complement-cum-antidote to orthodox texts, with the aim of synthesizing that from prospective looks in across a scattered literature tools and advance warnings analysts may not have received form their courses. It particular at what we can call the sociology and the pathology of assessment. 1.2 Limits to determinate assessment: the scope for judgement, bargaining and massage It can be suggested that orthodox planning literature and training has had a set of intellectual and socio-political assumptions which render assessment determinate impose and and orderly. Real world divergences from those assumptions limits to determinate assessment and introduce elements of judgement improvisation, indeterminacy and controversy. problems, confines analysis the necessity "to In trying to face actual broaden the scope of the analysis beyond the regarded as optimal within the operations research culture [means] finds itself persistently difficulties]" (A. Villiams 1972, 524). in the terrain [with greater -3- Cost-benefit analysis price from taken operations thing conclude volved the that in ideal. full the despite its use of a term like shadow theory, inevitably falls far short of the It can in practice almost never undertake any- procedure for deriving shadow prices. One may still types of simplification and corresponding judgement in- CBA are of the same order as in all social research that touches reality. So if they are not ruled out a priori one must try to judge their validity case-by-case conflict; while conflicting the example, optimization research like for (1) this is where judgements will often even if exhaustive analysis were possible, uncertainty and judgements world But would remain about future conditions, especially in markets from which recent CBA methods have to derive more objec- tive guidance. Evidence on sparse. Its relative shortage might suggest a disturbingly ritualistic use of the studies. quality and consistency Some of the work available suggests frequent marked variation is assessments reached even objectives and methods.(2) It was claimed analysis alent the in for the the by analysts thought to be applying the same competing OECD and UNIDO methods of cost-benefit 70's that examination showed them largely formally equiv- and that in practice they gave similar results. hands Mirrlees of the method, sensitive the alternatives. In very clear amount 1987, of from experts questions analyst. Reviewing studies that used the Little- (1978) argued that their results were often less matters, the But this was only in notably practical the specification of relevant policy application of those methods it "becomes careful reading ... [that] studies incorporate an enormous judgement 158).(3) mysterious. same Stewart to and of assessments has been rather and Such intuition on the part of evaluator ... " (Stewart differences in assessment often remain tacit and The study by Margolis (1973) concluded that U.S. social policy rarely were differed important extrapolation of on "basic" theory or method, but instead on what and how to formulate them, and on interpretation results. One must then question the sufficiency -4- (sometimes common. Further even the return the Yorld of later estimates deeper doubts of the "basic" techniques affirmed in Bank's published figures (e.g. in IDA 1982) of rates of calculated years significance) on appraisal, on project operation, show (notably about ｣ｯｭｰｬｾｴｩｮ＠ and again after some a consistent excess of appraisal estimates over for rural development projects). This raises the security/manipulability of assessments as used by both applicants and agencies in their quests for funds and for disbursement. Yithin the area of bargaining, the scope for judgement merges into scope for "massage", the discreet squeezing of materials to give a desired appearance. Section 3 of this paper offers a brief tour. "Massage" of but or in Stern (1976)'s phrase, "SOSIPing" - Sophistical Obfuscation Self-Interest not from and Prejudice - is absent from most training literature, practising Indeed in that context it far assessment studies SOSIP, or instead limit, contain and rationalize do decisions? limits may tradesmen's informal craft lore. be de rigueur to show a man-of-the-world knowingness. That on not question only should study How be considered within the real world of resources but also on study motives; in other words, with reference to the sociology of assessment. 1.3 The need for some sociology of assessment Socio-poli tical literature. concerning It the look at this open systems, assumptions is worth are usually left implicit in orthodox planning drawing out some "social aspects of planning" planners and not only, as usual, the planned. One needs to terrain of judgements not only in terms of planning theory complexity, control areas, "logical frameworks" and so on - but alsoO in the terms of the workday tradesman world: role and professional conflict, scope for different organizational loyalty, political judgement and manoeuvre. For judgement provides scope for differing perspectives, reflecting institutional and professional settings and influences. -5- Understanding the use of assessment methods then involves "the painful of observing people" (Simon & Stedry 1969, 272), uncomfortable as necessity that may be for the economic theorist (and for governments: Smith, 1985). Gurel to for example referred to "complications still inadequately covered ••• do not with technical and methodological issues, but with the organiza- tional context .•• and most of all [the divergent characteristics of program manager and program evaluator and their interactions which influence program evaluation 13). its efforts [and] most centrally affect the outcome .•• " (1975, 12- These interactions are also important in ex ante appraisal, if not in abstracted literature. The ex post evaluators are naturally closer to reality. Section 2 introduces a set of very basic themes and contrasts in the sociology of project analysis, which any student should encounter. Planning is not disembodied technique. It is attempted by particular people, potentially affecting others. Each have their particular ideas and interests. Planning theory and "the theory and then fails if it is merely optimization theory or systems is not also administrative theory, alive to administrative art realities. Yithin this area, I agree with Yilliamson (1982, 585) that most important unsolved problem of the [administration] literature ... [concerns] bureaucratic organizational subgoal pathologies pursuit," [i.e.] and so on. goal displacement, This is why Sections 2 and 3's look at the real world will often emphasize the underworld. 1.4 Coverage and objectives The issues now introduced apply equally to appraisal and evaluation, so both will and be covered. inter-group "programme" They involve similar inferential logic, conceptual issues relations. Also the issues apply to both "project" and levels, and indeed to much of planning and policy-making, there being no very sharp division here. -6- Experience from DCs reasons that the literature DC Adding well approaches is as LDCs will be referred to. For the same in LDCs have been generally derived from DC work, fuller. But most examples will be LDC oriented. qualifications for LDC conditions is also less necessary when one is discussing theory, are as problems, given the generally weaker institutions, staffing and supervision. the available information, Of special interest though possible impacts and distortions in foreign donor financing, where resources may not have to be returned and/or the financiers are often eager to disburse. Some features of such aid can apply within many public sector, state socialist, or intra-organizational contexts; perhaps even to the 197481 outpouring of commercial bank loans that recycled OPEC surpluses. The transportation and especially concerning importance and areas, receive water or for more will provide most examples, This is justified in terms of their relative public They are among the foremost investment sector and foreign aid funds. Roads alone housing; and have been popular with both the spatial analysts and project analysts. and economic also "harder" benefits suggested Transport projects lend themselves well to financial analyses, compared say sectors public investment than agriculture or manufacturing or the 149). roads. technical interest. especially often construction to that compared say telecommunications (Le. more to health and education projects. or domestic water supply, they offer quantifiable in money terms). It has been this allows them to receive excessive attention (Self 1975, For nor are transport projects (and especially roads) short, compared to mining, on vague but appealing "social" goals (mass access, national integration) or assessment. Further shielded supposed unlike benefits commercial which projects, might be played up in roads are relatively from criticism if selection and appraisal prove later to have been excessively favourable. operating budgets. They do not make an open loss, or have large ownerThere disproportionate share plementation relatively attractive indirect is in of disbursing is then investment. a They straightforward, budgets (e.g. aid). danger that roads claim a are expensive but their imand both features can be One can go on building more -7- roads almost indefinitely, and politicians, and electorates. in offering a wide they are popular with each of donors, Finally, roads are also of technical interest range of ready alternatives for analysis: alternative routes, design standards, methods of construction and management and labour supply, alternative timetables for upgrading. This potential may not however be sufficiently used. It is have also felt though that most of the sectorally specific examples will some wider relevance. the reader can consider this for his or her own particular sector and setting. To review objectives: - the general objective was stated at the outset, i.e. to consolidate some of the scattered written and verbal material on project assessment as it is seen in reality. It is hoped this will provide a suc- cessful reference for teachers and students. Secondly are: and social manipulations more specifically, the real world dimensions to be considered interactions in project assessment, in Section 2; and possible of the elements in assessment, as in Section 3. Section 2 on sociology is the more important, considering the motivations and perceptions behind particular possibilities. tactics, for which Section 3 only introduces some of the Similarly while the discussion of tactics will be a little more formal, priority should go to Section 2/s establishing of awareness of the possibility and directions of tactics, rather than to those areas which lend themselves Tactics many more easily to formal sophistication (cf. Laver 1978) (4). in practice can often be crude, yet still influential, and also via channels besides the more scholastic ones. become more Vhile the.sociology should rigorous - especially as and when it receives increased atten- tion - the issues raised need to be looked at in training already. Thirdly, flagging some further issues that arise, as in Section 4. These are the lowest priorities - and in particular the issue of what is "the" net -8- outcome of the various themes and tactics is also partly misconceived - but they help to situate and round-off the discussion. 2. THEMES; DOERS AND CHECKERS - ON THE SOCIOLOGY OF PROJECT ANALYSIS 2.1 Some roles and contrasts In our we must bear in mind the presence of several different agents; for example: (i) look at bargaining between resource-controllers and resource-users, executive executive departments, at lower levels and (ii) at their HOs; (iii) Ministry HOs; (iv) the Finance/Planning Ministry; and (v) foreign aid organizations, at local office level, and (vi) at HO level. and contractors Bargaining between Consultants can come in to advise or take on the work of any of these. can occur adjecent between pairs. more or less any pair, though it is usually Many other types of stage can be sketched; but we will concentrate on general themes. The generic role contrast to be considered is between those directly in- volved in and responsible for an activity, and those involved at one remove, responsible for its assessment. evaluators, Gurel course also can (1975) apply of Suchman (1967) speaks of practitioners and managers and evaluators; and the contrast of in ex ante appraisal. There are also some related contrasts. One is between economists and engineers, as discussed by Boulding (1964). Indeed some of the discussion that follows grew out of considering relations between and economists builders. important subject than to and engineers, as well as physical planners, architects But I believe that most is of wider relevance. training; assessment) economists-turned-managers and (or engineers-turned-evaluators role, not background, as Gurel notes. Role is more when themselves act primarily by -9- The second related organization internal (see contrast 2.2. assessors context. below). may A technical is between those inside and those outside an be But in principle, if rarely in practice, detached; department's and what is "inside" varies with headquarters internally assessing the work of one component of the department will act in the evaluator style; and while another ment,it may assessment part become is of Government may act in that style toward the departpart made, say of the internal camp when by a foreign aid donor. (1) a non-government So, roles vary with case; and role tendencies are tendencies, not necessities. Gurel (1975) provides a useful stereotypical set of contrasts, which can be tabulated as follows: Items and 1), 2), 3), and 5) relate especially to the contrast between insiders outsiders, contrasts and which this section looks at next. It then goes on to the between doers and deliberators, and engineers and economists (2.3 2.4 below). Finally it looks at overall relations between checkers and those being checked, between those administratively allocating resources and those claiming insider/outsider them and (2.5 below). Yithin doer/deliberator this contrasts bargaining can take context the on additional interesting features. Manager (Practitioner) Evaluator (Assessor) 1) Committed to programme, its survival and stability. Sceptical about programme; committed to progressive change. 2) Ambivalent to evaluation. Committed to evaluation. 3) "Organization man", "on the team". Individualist; refers rather to external peer groups. 4) Yants fast results ("satisfying"), Yants to do thorough job ("research"), 5) under his control, for his purposes. Hates "whistle- blowers", i.e. publicizers of criticism. with recognition/publication. 6) Doer Deliberator -10- 2.2 "Insiders" and "Outsiders" To start with the relatively familiar: "insiders," if we follow Gurel, tend to be committed assessed. of to their Organization programme and ambivalent towards its being theory discusses "organizational ends", i.e. goals the organization's perpetuation, stability and expansion, whether or not justifiable in terms displacement". maintenance "sunk" emotional avowed Organization and (i.e. of growth general objectives. members' and This is "goal career prospects are related to the of the organization. nonconvertible) commitments, more They are also likely to have personal "investments" in the organization investments in personal relations not easily replaced. Similarly, promotion is more likely to go to the safe member solidly committed to organization perpetuation.(2) Confidence and is the value of one's work and one's organization is also craved sought. (1972) for Likewise the example has feeling of security through solidarity. hypothesised as common the Janis phenomenon of "groupthink": a mutually re-inforcing consensus in face of the cognitive and ethical uncertainties confirmed and stresses in action and decision. Each feels by others' apparent assurance, self-censors his occasional "weak" disloyal doubts, and thus generally participates in a cohesive, stress- free, harmonious suggested banks, and and in such "team a herd-like, spirit". pattern in Anthony Sampson and J.K. Galbraith have the optimistic pouring of funds by private to some LDCs in the 1970's, and their neglect of others; similar earlier cases in banking history. "If they're doing it too it must be OK". Ye thus have two levels of goals typical in organizations' self-presentation to others, flicts splendid and and to some degree to themselves. responding objectives. evaluation" to the Patton performance (1978) refers Firstly, papering over conethic, to the there are broad vague "charity model of as widespread, with self-assessment according to the quality of -11- intentions and quantity of effort. The quality of intentions in fact becomes assumed, as part of a self-referential idealization. Secondly, the specifics of proposed or - especially - actual activity, argued to be merely the implication of, or necessary condition for, those vague splendid objectives. This is the brandishing of quantity of effort. Fixed is the commitment to the organization, clothed by its fine objectives and vague rationale. Fixed too is the commitment to present organizational activities, when these do not jeopardize the first commitment. But when performance appears unsatisfactory and the organization is under threat, evolution may occur under the guise of "clarification" of what objectives "really are" (and "always were"). Being ideological, goal language is not binding, and the "goals clarification game" may proceed. Drawing on the large body of experiences in the goal-based evaluation research of the past generation in North America, Patton (1978) describes "the goals clarification game". I have restated the game diagramatically (see below) . ...---____ Organization staff are asked to provide goal statements _-----. t Evaluator judges whether these are 'real' goals' i.e. clear, specific, measurable, etc ＱＭＴｎｯｾ｣｡ｵｳ･＠ t Staff give up evaluator writes the goals of 'conflict configuration' ) ｾｅｶ｡ｬｵｴｯｲ＠ Yes gives up; Accepts non-'real' goals Evaluator 'wins' ｾＭ Later: 'fuzziness gambit' by staff Later: 'goals clarHication shuffle' by staff Evaluator gives up; staff 'win' ＭｾＮ＠ -12- The game starts representatives with outside evaluators pressing organization members and for statements evaluators becoming "conflict configuration", programme of dissatisfied objectives. i.e. their objectives. One even eventually giving or intra-organizational reason up, disagreement for is a over Goal specifications may still be made and used, but have limited power. After particular goals have been specified, whether by evaluators or staff; typically after especially it has evaluators have committed themselves to a research design; after the research is in fact completed and most typically when produced negative findings: the game is likely to continue with one of: (i) the that "goals the sentation clarification goals of shuffle": the organization now "realizes" they earlier specified or accepted are not adequate repre- their "true" goals. Goal priorities are reordered and/or additional goals are "remembered". (ii) "the fuzziness gambit": an organization previously incapable of stating clear operational time goals and their relative priorities, now for the first "realizes" what its goals are, though regrettably these are other than those which evaluators had of necessity to specify by themselves. Regardless of whether such "moves" are opportunistically motivated or are legitimate responses to difficulties in the notion of organization research conclusions. The three problems now mentioned (internal conflict; legitimate adjustment; opportunistic Firstly, plural (As we evasion) represent the three main questions about objectives. internal relations: whether and when one can treat internally agencies as united around clear and consistent programme objectives. will organization see, there may however be some cohesion around objectives of perpetuation.) Secondly, external conditions and internal -13- expression: to consistent thirdly, tions what extent are even unitary agencies capable of clear and objectives in face of very complex and changeable environments? external expression: the functions of objectives in agencies rela- to other condition for agencies. obtaining Stating objectives resources. of certain types may be a Being judged by objectives which have not been achieved may induce clouding or revision of stated objectives. Insiders' ambivalence cooperation. assessment can thus sometimes lead to non- This can be effective where outside capacities for monitoring, investigation outsiders towards and rearrangement are very limited. are more powerful, is to try to co-opt them. non-cooperation may be An alternative, where And much worse than in store for insiders who become "whistle blowers" and reveal internal failures and scandals to outsiders. Having looked orientations, at organizational before turning in loyalty, 2.4 to we the move on next to executors' orientations of assessors detached from the executive organizations and from direct action. 2.3 "Doers" Technical to practitioners commonly claim that with the time and money devoted refinements Delays' of project justification they could do work of more worth. impact on money costs is also typically stressed by non-economists; though public revenues might also be inflating, and the optimal project start-date is often not immediate. Running deeper "doers" and "hypothesizers"; between "the engineer" concerned with specific actual of or is" Veblen's freed of these complaints is a temperamental clash between imminent achievements, and "the economist", talking abstractedly impacts "what beneath on the against vision carping whole "what society, including under notional alternatives: is not" (Boulding 1964). Boulding refers to in "The Engineers and the Price System" (1921), of a world restraining pen-pushers, where engineers would solve all -14- problems. (There Stalinist views engineers to is of close a the control analogy triviality contemporaneous to Leninist and of allocative problems, which allowed and left economists to justify History and planning The doer may further sense that extended consideration undermines decisiveness. Also involved is a clash over professional territory; much of State.) the work of the evaluator appears as a questioning of the autonomy of established professions (to dispose of other people's resources). Vhether consequent or not on these deeper clashes, doers' complaints must be considered on their merits. told one the experienced same One variant is that the practitioner could have thing without a costly study. (3) engineer travel For example: "Let an a road and know its average daily traffic and he'll tell you directly what sort of improvements are justified. international analyst. Forget the Vhat the engineer actually does is apply rules-of- thumb developed from samples and calculations by engineers and economists to cover cases too small to justify individual consideration. be disputed materials just and considerations how small economic there is too small. Of course it can The more variable are local prospects, the more route alternatives and network are, and so on, the less adequate are rules of thumb: except to select roads for further study. The main variant of the objection - in logic and perhaps intent - is instead this: little credence is given to the economists' methodology aiming to cover these case variables. The engineer is familiar with the broad use of judgements in his own project studies not - which are literally "feasibility studies", of controlled effects, essays in fine normative distinctions. something, more tempts projecting with at whether to do it. He may be sceptical over all at- uncontrolled variables; and may well have experience formats of cost-benefit analysis, as the conventional penultimate the chapter than For him the study is how to do in his feasibility study. Indeed much of transport project economics and sister branches was initially developed by engineers who were asked justify to competent, investment choices. Feeling both cautious and already the engineer is likely to be sceptical of new entrants with more -15- sophisticated methods, essaying not just feasibility but efficiency, even optimality. Transport project textbooks and potential importance. frequently at this ordinary working level, outside the superstudies, often excludes many issues of prima facie the high conventions. ' deSlre economics Self (1975, estimated rates 55) suspects that transport projects' of return reflect peculiar internal In many cases though, these conventions seem to reflect more a f or . 1"lClty, (4) Slmp E ' nglneers a 1so kn ow ' ca 1 cu 1atlons ' t h at t helr underlying vehicle operating cost figures are only rough approximations, and so often resist refinements elsewhere in the analysis. is to ensure that, even selection that simplifications are not opportunistic or arbitrary; and if all biases between sectors. discussing projects are not But simplification remains essential, as will be seen when "framing" over are assessed on this same basis, significant introduced between types of project, within and (3.3 below. alternatives are considered. clashes Economists' concern Part of that discussion concerns which Ye can look next at engineers' and economists' attention to alternatives and the balancing of simplicity and significance. Neglect e.g. of the directly 1981 adequately is need narrowing economic available USAIO review technical alternatives appears widespread, of rural roads projects found "a failure to consider alternative design standards" (Hicks 1982, 29). There to keep assessment manageable, but this should be by a progressive down of alternatives, using if necessary fairly rough and ready appraisals at earlier stages: not by initial arbitrary restriction of focus, followed by belated economic refinements. In LOe road programmes for example there is a considerable range of alternative often road standards worthy of consideration. obtainable construction full and through improved Also higher returns are quite maintenance, spot improvements, stage the use of temporary structures, rather than conventional scale upgrading. But as an engineering professor put it: "engineering -16- structures are satisfaction interested monuments in in a to their builders, completed project. and there is much personal That engineers should be especially large projects is only natural and it seems likely that very large projects are particularly susceptible to premature fixation" of design concept (Linsley 1964, 26). Practitioners' concern tunity costs can obvious and which weight be over a the visible rather than with notional oppor- politically sound: "a small benefit which is clearly goes politically diffused with to a politically powerful group is of much more than a larger benefit which is not visible and which is larger and politically inactive and impotent section of society" (Boulding 1964). Limits of time and information in design contribute to conservatism, of two sorts. Firstly, towards overbuilding, to minimize possible later criticisms arising from whatever minor problems in operation (Linsley 1964); secondly towards the engineer happens to be familiar with and trained in. Thus one sector survey comments repeatedly on uneconomic "accepted practice" in rural road design been questioned colonial (ODM 1978). in Imitation of DC design concepts and details has many ILO studies, commenting on the colonial and post- patterns of reproduction (e.g. Irvin 1975). It is worth adding to them on possible perceptual and professional patterns. Objectification of practical attitudes is sometimes seen amongst technical men: marriage to a particular practice or set of technical standards as "the proper way", for building a house or whatever. Supporting reasons will be provided, but conditionality upon economic or cultural variables may simply not been have understanding position, reproducing standards". considered. the e.g. reasons the honoured By for expatriate imported undermining familiar rules learnt without them, or it may jeopardize the technician's settler technician in an LDC, doggedly specifications and resisting "lowering of -17- Full professional alternatives of role "the and costs money, to multiplicities of functions and criteria. against proper" cut engineers and architects are more attuned to analysis of ask outside intrusions may sometimes reproduce this fetish of artefact. by An architect irritated by economists' attempts to questioning for design features the judges of good buildings,,(5). for an occasional safety, is always and the case to may say: "If you want to save fewer building, not bad buildings", and implicitly "and we are comfort, But defence prestige toy) However building designs never (save maximize fulfilment of design values - flexibility, etc. - either singly or jointly. Fulfilment limited in view of costs, both the impact on other design values resource case, cost. Balancing of these multiple values will vary from and can involve clients, architects, economists and others. It cannot be architects' exclusive preserve. in exaggerating the The fallacy of technocracy is importance of purely technical factors, or making the implicit assumption of unquestioned values (Self 1975, 3). The contribution of economists and trained evaluators derives in principle from the prominence in their training of a systematic balancing oaf alternative values; balancing and say a in having more to offer than a single sector expert when building's functional values against the value of money, based on greater awareness of the value of that money used instead for say a road or borehole. involved to here too is a struggle for professional territory, and resistance restrictions preference nical Economists' intrusions are not always well directed; but on the preferred expression of acquired skills. for alternatives offering the most professional challenge, tech- practitioners engages pression their of Evaluators' possible, Besides a do skills. skills, familiar including enjoy choosing between alternatives when that too But if it becomes more a constraint on than an exand imposed by others, then it may be resisted. resort is professional to ask as many technical men's opinion as outsiders and mavericks, to check that professional convenience has not masqueraded as professional authority. -18- Neglect of alternatives arises in wider ways than in choice of engineering methods or design for a specified facility. good way of pursuing more general purposes? Is the specified facility a Even the famous London Victoria Line and Third Airport studies both in effect begged the question of whether the pre-specified rural road praisals, form of evaluations they alternative still ways investment found that usually of was appropriate. though failed to of Hill's review of wider scope than most ap- take a broad fulfilling more general goals. perspective of alternatives to rural roads sometimes include even villagization, as well as heavy-duty and fourwheel-drive vehicles. Often the greatest waste occurs not in doing things inefficiently but in doing things that were unnecessary. this questioning of functionality with respect to deeper purposes obviously not only intrudes on the tial as autonomy of a practitioner and organization but also carries poten- threat to their whole operation. an 'objective investigation' "What the social scientist thinks of the practitioner often takes as 'hostile attack' (Wilensky & Lebeaux 1958, 20). It is important after this to also put the evaluators/economists into some professional perspective. 2.4 "Deliberators" Each profession specialism and tends its to feel (great) it alone can properly appreciate its own importance. This appreciation then validates professionals' drive for activity, power and the expression of their skills. Physical planners central; road development; have engineers economists analysis, and often economic tests or should be a holistic tend to conception stress with the physical planner roads as preconditions for all stress macro-economic consistency and cost-benefit tend to yardsticks "the belief that there exist fundamental according to which policy decisions can and made" (Self 1975, 5) - "econocracy". Physical planners may com- bine engineers' physicalist biases with economists' society-wide conceptions and claim to preeminence. -19- The appraiser/evaluator ("assessor") as a distinctive actor stands separate from the practitioner, the doer. existing proposal negativity. run have or into serious are often resources available. titioners tend of no his past experience. This gives a proneness to First, ex post evaluations often are undertaken when programmes appraisals saying a His role is mainly reactive, to look at an to problems in a context Second, sell (e.g. exploratory new programmes); while the of a mass of proposals exceeding the assessor will quickly learn that prac- new programmes and defend exisiting ones. Third, is the assessor's distinctive contribution, visible legitimation separate role. The stance-apart may also have some unfortunate consequences in the way the assessment problem is conceived. One is the weakness alternatives. analysis of In in practice, principle, alternatives, outside economists text might books, in considering hold a vested interest in that could balance technical men's learnings to large sophisticated designs (or whatever else is fashionable). But in practice, economists may be "pretty useless on the whole in their contribution to the identification of relevant [alternatives]. policy-making and some is other concerned tremendously planners with weak on design" (Joy 1973). typically line urban the of technical alternatives design, or So what is going on? lack the knowledge to put forward technical options, while men are already too busy pushing through established options trying economists of themselves as distinctively exculpation is that economists/planners (see Alexander 1982 on planners) without Yet economists analysis of alternatives, as "primarily to do with decision making or choosing (Thornton 1973, 3). On think Planning and out may the unproven remain range ones (Hirschman 1967). though even when Neglect of technical there is no pressure; and for of alternatives is wider than choices of engineering e.g. it includes not doing the project or doing it later or smaller bigger or in stages, or instead doing another project. Yet neglect of -20- such comparisons too seems to have been widespread (Joy/ODA, on water planning). Generation of alternatives may be less congenial work than the sophisticated design or lines. Another possible reason is a planning process where the economist is only "called in of as a he project along familiar inhereted disciplinary usually is - to provide an [appraisal] after a has been outlined" (Joy/ODA); (emphasis added), e.g. when it is time scheme to assessment sell merely the to taken scheme provide at to a donor. elaborate "[All] too often [the economist] is used justification for a set of decisions already technical design stage" (Irvin 1978; similarly Horgan 1983 on the use of social analysts). Behind case these features lie more economists like theology than engineering, according to Peter Self. did later turn allocation-amongst-given tastes, technologies remoulded. tion, the were, assisted organizations brought without had a fancy logic of a comparable feel for how been moulded and might be did their training and experience cover organiza- and politics. need [is] sociologists if other engineers politicians, the and they But despite this, economists felt that for economists to take the lead '0' being technical specialists of all kinds ... [such as] engineers, .. agronomists surprising planning Vhen business of making choices [is] the centre of the universe as the by to possibilities, often, Nor, implementation "[since] only In the of economics this includes a long history of rationalization after the event, it patterns of disciplinary organization. who economists' or in and professions physical so on" (Thornton 1973, 3). It is hardly take a different view of the matter: not planners, but also administrators and some economists' conception may only provide - as per requirements, and perhaps even without their action - some of the data on which their ordering logic then gets to work! The next effects. danger of the stance apart is a possible narrowness in envisaging an honoured lament amongst engineers is that if Columbus had had -21- to produce a feasibility discovered. The feasibility study, typically real study assessors' a then the Americas would never have been reply is that Columbus' voyage was itself the favoured category amongst economists, since it is low cost even if high risk, and provides information. The doers' point however concerns ability to envisage possible outcomes - such as finding the Americas. highly than lives. Vithout denying this danger, different men-of-the-world often "feel" the differently, and Here they value their man-of-the-world "feel" more careful the but narrow speculations of those with restricted goal is to bring many types of "feel" together for rigorous cross-examination. Vhile practical men may feel that an assessment is negatively biased they sometimes are unable to express precisely how. and precisely argued by various evaluators that the mass of U.S. programme evaluations in the 1960's arguments apply more widely. Firstly, However it has been strongly and 70's was negatively biased. Some of the there was a preoccupation with avoiding Type II errors, the incor- rect acceptance of weak programmes. if it is not consciously chosen. This weighting can be an erroneous bias External funders' possible assumption that the onus of proof lies on programmes to show impact beyond reasonable doubt can bring a programmes. ment neglect of Type I errors, the incorrect rejection of good The same assumption is seen in general evaluations of develop- assistance, and can affect project appraisal too (e.g. if in a bureaucracy the priority is not to make visible mistakes). Next, it has been argued that several features of orthodox evaluation con- spired against refuting the null hypothesis. programmes differences was for fail to refute it: "[it] is very hard to find significant anything ... " (Arrow 1976, 52). that state activism was misguided. case-by-case piled did up. excuses or Certainly most U.S. government The conservative response Programme defenders of course found replies, yet lost credibility as similar findings But some analysts began to react as Arrow: " ... 1 am beginning to -22- question the methodology; I cannot believe none of these things matter" (loc. cit). The second their argument then was that orthodox preoccupations with outputs and statistical correlation with inputs had diverted attention from whether measures had been properly implemented. In many cases they had not. This tells against a programme if it shows it is hard to implement; but that is a separate Inattention issue may plementation implementation reflect can may simply have been botched. the influence of research station work, where im- be taken for granted. (This has had an opposite influence in project appraisal, i.e. producing over-positive conclusions.) Thirdly, that with programme operation left as a "black-box", attention had diverted been from what in detail was happening and why: output failures were noted but not understood, hardly a basis for drawing conclusions (Cline 1976). with Relatedly, large emphasis samples on statistical significance brought a concern and a neglect of "maverick" cases. With numerous sig- nificant factors involved, the combination of factors underlying the various maverick successful cases may - in the absence of policy - be rare, and they may be swamped by the mass of mediocre or unsuccessful cases. is then sible no Overall there significant impact, but instead of studying successes for pos- lessons the normal tendency of "burying material that does not find a natural place in a coherent account" (Ravetz 1973, 225) takes over. "Mavericks" might be non-replicable freaks; but they might not. A fourth (Scriven bias 1971, links to the Deutscher emphasis 1976). types of effect, a priority specified objectives Having spelled out intended impacts the evaluator goes out to look for them. other on neglect Bias arises if there is a blindness to of the unintended and unexpected. touched on the Columbus case in ex ante appraisal. either positive or negative. This bias however can be On the other hand, experience should bring learning, and consequently both revised goals and improved performance. if previous goals We But are enforced in evaluation then there may be a negative -23- bias, i.e. other hand, since men notional assessing people by goals they were no longer pursuing. On the unforseen side effects are perhaps more likely to be negative, may congenitally fulfilment of underestimate their wishes. problems, preoccupied with the Reference only to ex ante goals and failure to look at side effects here give a positive bias, associated with a co-opting induction into practitioners' social circle and circle of ideas. Taking all four factors into account the average resultant was very probably negative for external ex post evaluations. in-cheek two "Laws The second of Yilson's tongue- of Evaluation" thus states: No programmes are found to produce the desired effect (at least when studies proceed independently of the executor agency, etc.). (6) 2.5 Claimers and Checkers - negotiation in administrative allocation Having looked at the dimensions of "insiders"/"outsiders" and "engineers"/"evaluators" , we can look now at overall interactions within the administrative allocation of resources. Yithin this setting, one strand of practitioner-assessor conflict is simply dislike of being disagreed with. But further, the practitioner may particularly dislike criticism from an outsider, someone not as deeply involved in the specific case, perhaps with a different training. to self-esteem. role is such deal with unit This is an insult - and, when correct, also an insult the assessor may, worse still, be of low rank. that even the low ranking may wield power. heads, busy men Assessors' So commonly they who yet feel obliged to front for their organization when it is being weighed up by a junior outsider. Another strand (usually) "Yell, Besides is unstated if I'm resentment at complementary "not being trusted", a common cry, with clauses like "And after all I've done", not appreciated •.• ", and "Then you can do my work instead". the real fear when something is being concealed, the ordinary over- involvement in role and the insecurity of the "child" within us, this line of resentful defence does relate to a significant issue: some trust is vital -24- in any organization's operation; how far can it then be relied upon? is a use trade-off of There between the salutary requirement for demonstration and the resources in meeting that requirement. In principle if decisions have been taken in a systematic way then there should already exist a statement on record of the considerations taken into account, available for easy reference when an analyst seeks to assess a proposal. building design there should E.g. for a proposed be a formal statement of the presumed staff numbers, units and functions, including as they might change over the building's life-time. they were mislaid. them never Even in In practice such statements may not be available: because the prepared, when or just casually guesstimated, or possibly they exist there is often work involved in presenting fashion required by the assessor, which makes practitioners curse at "unnecessary" work. Lessons for assessors here include: sometimes use "spot checks" rather than blanket investigation; make sure practitioners are aware from the outset of the exact content and format of information they will need to provide later; and possibly help use practitioners retain a distance a division of labour between more internal assessors who collect information, and more external assessors who and authority which reduce the occurrence of the inter- colleague call to trust. The to limits to trust arise in familiar ways. allow exaggerate smooth the To acquire the reserves needed functioning in the face of uncertainty, organizations may resources required in a programme, and hence exaggerate expected benefits (Caiden and Vildavsky 1974, 63). assessor may have to mistrust all. Unable to trust all, the Even the trustworthy organization then has three inducements to become a "games-player": pique, expectation that an assessor they are will assume exaggeration and cut back claims regardless of whether exaggerated or not, so that it's rational to exaggerate; and lack of assurance too that when one is honest then other claimants will also be. -25- Mistrust of other claimants may extend to their whole area of endeavour. Each specialism is commonly gleefully aware and impacts of others. Other pretentious, or at less best familiar methods and titioner feels that Practitioners "something" costs (including must and be other practices. specialisms something than (e.g. be seen as spurious and one's own specialism, where the prac- cost-benefit better costs in least of may certain in their contribution than are the practices at f the questionable foundations 0 of value is being done. analysis) do commonly feel that "nothing", Le. they ignore opportunity maintenance, cultural costs, etc.) of their Even when aware of opportunity costs they may downgrade them in the way just outlined, other claimants being suspect to them. Something similar ticular sector. [like only may Self argues that economists of education tend "to be biased other specialised analysts] towards justifying economically [i.e. not on other gorounds] the service they are studying or the organization they serve" (1975, 81). whichever move apply to evaluators too long associated with a par- sector from one One need not even posit specialists' induction into they happen to land. Also possible is that some analysts sector to another until they find one to emotionally devote themselves to. Amusingly enough though, while congenitally sceptical of the value of alternative to activities when competitive with their own, each department is happy accept others' rhetoric when it supports theirs. disbelieve A department may quite anothers' claims but as long as a Minister or an official publi- cation has made them, it is hard for an assessor elsewhere in government to reject them services when cited by the department. may disseminating huge trary potential to certainly all cite have a claimed that super-attractive marketing, probabilty it when For example, agricultural field they are in the process of producing and cultivation package, which will lead to justifying road improvements. and past pressing experience, for This may be con- but roads engineers will their projects. Conversely, roads engineers may claim that their own new technical package, say of on-the-spot -26- identification-and-design of more selective improvements for rural roads, to new custom-built standards, will halve vehicle operating costs, for modest capital and parable unfulfilled use this maintenance new commitments. promises While experience may indicate com- in the past, agriculturalists will certainly one to arguably tip the balance in favour of their new rural development model, and so on. This leads to the next strand in practitioner-assessor conflict, which was introduced earlier. Practitioners may feel that socio-economic evaluation is all a game anyway, a more or less arbitrary cover for one group's preferences being imposed over another's. Technical and practical men may contrast their long exacting training and tangible vocational spheres with the personnel and practice of socio-economic assessment: in their eyes often transient figures and fanciful generalists, with little hard theory, plucking from the air for "social values" of savings, effort, time, or life. this may occur within a wider syndrome of positivistic confusion on values, with self-licensing is a beliefs such as: "its all just opinion", or: "politics dirty game; so, it doesn't matter if I follow suit", and a lapse into cynicism. The final strand I will suggest is that resentment of evaluators can then derive from a feeling that they adopt a "holier than thou", superior, headsI-win-and-tails-you-Iose position, judging others. it is because If a project is approved of what they have identified in it, if it is rejected it is because of what practitioners have put and failed to put into it. This sort of feeling is always likely to attach to those who have the job of comparative assessors the assessment. lapse into self-regarding idealization, themselves cushioned from difficulties and responsibilities of action. "problem of marginalizes There But it can be fanned and partly justified when the is divergent planners" (Sen 1970). and This is an example of the A central planning body which antagonizes the non-central bodies will run into trouble. no substitute for trying to motivate all parts of the system with -27- common norms, so that socio-economic criteria are requirements accepted and by used the externally this, at non-central a late bodies in their work, not simply foisted on them stage. Sectional foci and interests set limits to as do limited capacities, so external assessment remains vital. Both strands are needed. To conclude: loyalties of internal strong 24). there are limits to what can be expected from the wider claimant organizations, or from the revelations of individual muckrakers ("a profession with an extremely limited future" so are the pressures and sanctions against whistleblowing; Branch 1971, Formal assessors may sometimes "join the team", even when not employed by the assessed organization - not only through social induction but because of assessors' own interests in not rocking the boat but instead maintaining a steady and predicted expenditure flow. Tendler This is the situation diagnosed by (1975) and Morgan (1983) for USAID project appraisals: "essentially an advocacy process" in which, as an AID review noted, "AID analysis in project papers has been overly and unnecessarily optimistic," but where "mission staff are rotated •.. often enough for most not to be around when impact evaluations are undertaken" (Morgan, 64-7). an exceptional case.(7) USAID are unlikely to be Section 3 dissects some of advocacy's tactics. 3. TACTICS; HOW TO SELL/SAVE/SLAY A PROGRAMME: SOME PATHOLOGIES OF PROJECT ANALYSIS 3.1 Occasions and Battlegrounds The battle to control assessment can start with rationalizations for avoid- ing having an appraisal or evaluation. ex ante appraisals may be resisted by reference to some preemptive claim: e.g. "the programme has such good intentions/is clearly socialist/has already been supported by the -28- President". Many others can be used to resist ex post evaluation; Table 1 gives Borgatta's sampling form the literature. They become rationalizations in being used only to avoid rather than inform assessment. Table 1: Lines of Defence1. "The effects of the program are long-range; thus the consequences cannot be measured in the immediate future. The effects are general rather than specific; thus, no single criterion can be utilized to evaluate the programme, and indeed, even using many measures would not really get at the complex general consequences intended. The results are small, but significant: thus, they cannot be measured effectively because instruments are not sufficiently sensitive. The effects are subtle, and circumstances may not be ordered appropriately to get at the qualities that are being changed. Experimental manipulation cannot be carried out because to withhold treatment from some person would not be fair." (Suchman 1967, 144) 2. 3. 4. 5. Secondly, when internally pseudo-appraisal/evaluation directed are assessments often the are undertaken, forms of result (Table 2). Next, when outside assessment is underway, attempted co-option or resistance may occur. Finally, when assessment gives unfavourable results, there may come rationalization of negative findings. Table 3 lists examples of re- evaluation. Again, arguments used may sometimes be apposite; it is in being resorted to that they become rationalization. Besides automatically rationalization there can be more vigorous rejections, with attack on asses- sors' rationality, function in several methods ways, and attitudes. Some of the arguments can e.g. stressing supposed long-term and indirect benefits can be part of rationalization of negative findings and of pseudoevaluation, as well as a way of trying to avoid evaluation studies. I should add as number 7: 'baptism'. USAID For example Lappe, 1980, alleges that programmes upped their distributive impact by redefinitions of 'small farmers', 'beneficiaries' (to cover all people in a project area) and 'basic needs programmes' (to include high cost roads and electrification). -29- Table 2: Examples of Pseudo-evaluation 1. 2. 3. 4. 5. 6. "Eye-wash an attempt to justify a weak or bad programme by deliberately selecting only those aspects that 'look good'. the objective of the evaluation is limited to those parts of the programme that appear successful. Yhite-wash an attempt to cover up programme failure or errors by avoiding any objective appraisal. A favourite device here is to solicit 'testimonials' which divert attention from the failure. Submarine an attempt to 'torpedo' or destroy a programme regardless of its worth in order to get rid of it. This often occurs in administrative clashes over power or prestige when opponents are 'sunk' along with their programmes. Posture an attempt to use evaluation as a 'gesture' of objectivity and to assume the pose of 'scientific' research. This 'looks good' to the public and is a sign of 'professional' status. Postponement - an attempt to delay needed action by pretending to seek 'the facts'. Evaluative research takes time and hopefully, the storm will blow over by the time the study is completed. Substitution an attempt to 'cloud over' or disguise failure in an essential part of the programme by shifting attention to some less relevant, but defensible, aspect of the programme." (Ibid., 143). Table 3: Rationalizations of Negative Findings 1. 2. 3. 4. 5. 6. "The effectiveness of the programme cannot really be judged because who could use the services most did not participate. Some of the persons who received the services improved greatly. Clearly, some of the persons who recovered could not have done so if they had not received attention. Some of the persons who most needed the program were actually in the control group. The fact that no difference was found between the persons receiving services and those not receiving services clearly indicates that the programme was not sufficiently intensive. More of the services are obviously required. Persons in the control group received other kinds of attention." (Suchman 1967, 144). "Find an expert to assure [people] that the matter is really in competent hands, .•• [usually possible] even when a project [has] stumbled into the most ghastly pitfalls" (Ravetz 1973, 348-9). -30- So, Vilson's found to jocular produce agency's "First Law of Evaluation" is that all programmes are the desired effects at least when one accepts "an own data about what it is doing and with what effect", and its own formulation of the evaluation, e.g. excluding "other variables that might account for the effect" and "a time frame (long or short) that maximizes the probability of observing [desired effects]" (J.Q. Vilson 1973, 133). Room for interpretation and dispute exists in many of the numerous stages in assessment: - choices of research design, statistical methods and analytical framework; interpretation assumptions tribution etc. of on of of contraints the statistics and setting of confidence limits; and technical and functional alternatives; dis- onus of proof; specification of objectives and criteria; Comparably, the analysis of a road project proceeds through discussion route methods growth alternatives, and of standards, normal "development a present and water traffic, availability, construction vehicle operating costs, likely traffic, generated traffic, diverted traffic, additional benefits": tender prices, weights on uncertainty and security; and much more. make materials Minor differences at a series of stages in an assessment can large difference to conclusions: firstly, when sub-conclusions com- bine geometrically - 1 to the 14the power is 1, but 1.1 to the 14th power is almost 4; subtotals, secondly, say costs when conclusions are based on the differences between and benefits, and hence vary disproportionately when subtotals are varied, the scope for "massage" is clear. Hanke and Valker for example review a body of literature in which: "Economists have long been critical of the apparent manipulation of benefitcost analysis They particularly cost analysis concern since roads, project planning and approval processes" in the U.S.A. discuss "water project proliferation, for which benefit- is only a mask" (1974, 329-30). Vhile my main examples will I too will focus on exaggeration of programmes' attraction, it is the main case. programmes of in "Torpedoing" - deliberate "sinking" of disliked - is also sometimes seen (Brewer 1981); indeed misrepresentation alternative possible programmes is a typical part of the selling or -31- defence of another. but sheer neglect is even more important than misrepresentation. The "torpedo" analogues to the cases below can easily be drawn out. Similarly appraisal may be more considered here than evaluation - for when programmes are already in place they may be able simply to ignore cri tieisms. There is a common feeling that "crime is not academically respectable" (Viles 1977 , 451). But that is one reason why "crime" needs attention: to counter an academic insularity that, after creating and diffusing complex techniques but not also understanding of how they may be abused, dismisses pathologies as of no theoretical interest. Awareness is needed for counteraction. One is not spreading dangerous knowledge: motivations to sell or sink projects generate these tactics without instruction, whereas it is more neutral parties who need protective tuition. One might distinguish between "forging", "fudging" and "framing". "Forging" the fixing of data sets is the resort of the cruder or desperate masseur. It has received increasing attention in even the natural sciences (Broad & Vade, 1985). adjustment It includes at the extreme, fabrication or favourable of data e.g. of present average daily traffic (ADT) flows. quietly it figures known More can include suppression of unfavourable information, and use of to be unreliable (e.g. ADT guesses by local residents; or suspected of being forged by others, say by delinquent traffic-count teams). Forging relies on the limits of supervisory capacity; or the ability to blame sudden changes in conditions, honest error, a wretched junior or a statistical quirk if the occasional supervisory check does give reliable opposed findings. But forging is not the main case: fudging and framing usually suffice, and we will concentrate on them. By "framing" I mean the issues involved in formulating the nature of the "decision-space": which variables to include; which to exclude; which effects to trace and how far; which contraints to assume and which to deny or ignore; which alternatives to consider (at least one - the base case - is -32- always implicitly considered). the critical importance of these issues is seen below in 3.3. By "fudging" I mean opportunistic use of the scope for discretion and interpretation methods seen in several stages. and assumptions programmes, which This scope is manifest from the range of adopted can then in be previous drawn on assessments according of comparable to convenience. Opportunistic choice of method - i.e. so as to give the desired conclusion is difficult to prove for a single assessment; one may need to see the assessors' treatment of a series of cases. But opportunistic application of method is easier to establish, since an assessment can involve many applications of a particular distribution) but may persisted tool. Tracing with criticize effects (e.g. on income be cut short where they are expected to be unfavourable, where favourable; a constraint may be assumed in one context but waived in another, and so on. 1976 of conventional project Carley 1980, Self 1975, and Stern appraisal practice as frequently ar- bitrary, inconsistent and opportunistic. "Fudging" can be partly unconscious. seeking some plausible guideline conclusion, in a sea when one is confident of a result, or of complexities and so hangs on to a one starts finding arguments for it and finding ways of terminating the discussion of others. In the following treatment of massage techniques we consider in turn: (i) data on demand and impact; (iv) description of criteria and and (v) synthesis of conclusions. benefi ts, exhaustive, base of of direct costs; (ii) "framing" of analysis of net impacts; (iii) analysis "costs" review and case and its subdivisions are not The review is illustrative not exact, e.g. division between "benefits" is a relative matter in assessment (relative to the chosen), their absolute levels. with one's real interest being their difference, not -33- The for listing of possible manoeuvres is oriented to the specialist reader, or classroom discussion. The general reader can dip into it, and refer back to it, according to his or her need and interest. 3.2 Minimization and "sinking" of costs Underestimation of direct financial cost abound, notwithstanding physical and price contingency provisions. The distribution of "errors" in cost estimation is not random, as commented on by Linsley 1964 and Hanke & Yalker Rondinelli 1976, Hill 1982, and Hicks 1982 all note this feature of 1974. road appraisals in particular.(l) Some methods are: (a) (b) (c) Omit consultants' fees from the cost-benefit calculations (seen in my own experience: "but this is our standard practice"). Gloss discussion of implementation and its difficulties. Play up the residual values of investments. And on costs more broadly: (d) Ignore inputs' opportunity cost (e.g. of technical and managerial (e) staff). Ignore the cost of necessary complementary inputs. These last two features have a long successful history in water control programmes (Hanke & Yalker 1974), and are alive and well in rural road and integrated rural development programmes (Hine 1982: 6, 13). Some aspects are illustrated in the Appendix. They lead into issues of specifying constraints and base case, linking up to the questions of framing, which in turn lead into one's overall appreciation of the problem area, one's situation diagnosis. 3.3 Framing the analysis Problem framing spans a set of vital issues: bounderies of analysis, the product and technique alternatives considered, the type of questions asked, -34- the constraints that are presumed and/or analysed. Sales-techniques here can include the following: (a) Trace effects to an extent (i.e. define problem bounderies) according with convenience. (b) Play up constraints project. Claim on the use of mobilization policy tools than this ｯｴｾ･ｲ＠ of unused resources (i.e. indirect "multiplier" effects) via use of this project/tool. (c) Assume complementary support; play inputs are irreversibly committed in down possible influence on those commitments ("the divide and rule" approach). (d) It • • • the value of the project .•.. depends on what would have hap- pened in the absence hypothetical state of of the affairs project. By its nature this is counter-factual and cannot be known; its definition and assessment must therefore be a matter of judgment" Stewart 1978, 158). So adopt a flattering base case where one has the choice; e.g. underestimate average yields before agricultural programmes (Heyer et aI, 1981). For roads assume that in the no-construct case much traffic will head down poor and rapidly deteriorating roads, implying great operating cost savings from new construction. (2) road will In general, assume that an unimproved be badly maintained (" no one's interested") and cause increasingly high traffic costs, while if improved it will be well maintained and allow consistently low costs. case is when another appendix). can be estimating costs from when estimating benefits. way of saying "sink" complementary costs; (This see Assume when convenient that the without-programme case represented by the before-programme case, attributing all improvements (e) Use a different base to the programme. Suggest that unfavourable assess- ments used misleading base cases. Emphasize or downplay obscurer political) according to convenience. constraints (administrative or -35- (f) (g) (h) Ignore or only downplay technical and functional alternatives; consider flattering alternatives (e.g. perhaps a no-programme case rather than the larger/smaller programme case). Lay the onus of proof on programme sceptics and proponents of alternatives. Choose a favourable time horizon (e.g. in appraisal, usually play down possible investment deterioration and include a long stream of benefi ts; post evaluation select a favourable in ex "representative" period). Why is "framing" a relatively underdeveloped area formally, leaving the informal massage sector considerable freedom? Firstly, orthodox applied research has concentrated on "problem solving", not problem formulation (Rein and Schon 1977). The problem is assumed to be either exhaustively specified and pursued, by the super-scientist; or, of more relevance, to be in some way given, as obvious, or by "the client", and/or by some "colleague". But in practical studies selection becomes critical: the identification of which activities and constraints are significant. Alexander (1982) links the relative neglect of the design stage to domination by the analytic thought style, whereas problem formulation, while involving analytic (and search) activities, critically requires non-analytic creative inputs. Secondly: "We are dealing with open systems, which in turn raises problems about defining the bounderies of the system (or of the problem space)" (A. Williams 1972, 552). The greater the scope the greater the indeterminateness. The need is for defensible principles of exclusion. These can be political, ethical, legal, administrative, economic. Self (1975, 54) defines this as the heart of policy analysis. Thirdly, not only constraint specification may be problematic because of the mass of technical and economic constraints but also of these possible legal, political, administrative and cultural constraints. -36- Fourthly, constraint specifications reflect both a theory of change and a value stance: for to call X a constraint means that the activity "changing X" is deemed to yield an inadequate rate of return, which judgement must depend on theories and values. These points are little more than a set of headings. To go into them would take us into the whole of policy analysis theory. Underlying "almost all disputes about cost-benefit analysis, or indeed about [normative] questions generally is the question of which constraints are to be taken as given" (Layard 1972, 11). They suffice though to suggest that framing is central, difficult, and prime territory for both dispute and massage. Just one point will be taken a little further. Fifthly, differences between assessments may reflect not only different principles of exclusion and different identifications of constraints, but also different responses to uncertainty about their standing and malleability, and different bargaining strategies. This can be illustrated for different school of cost-benefit analysis. The standard economic reasons for excluding various effects from costbenefit analysis (apart from their being of no interest) are that they are equally found in all projects, that they are achieved merely by division of resources already comparably well used elsewhere, or that they are better achieved by other means than project selection. the standard reasons for including a wide range of effects in LDC CBA are that there are unemployed resources which can be indirectly stimulated into action at no opportunity cost; that some indirect effects are vital for predicting project evolution; and that in practice some objectives cannot be pursued sufficiently by other policy tools. Past disputes in cost-benefit theory have often been implicitly about the constraints on other policy tools - about which variables are manipulable - "decision variables", and which must be accepted as given, "constraints". (Fitzgerald 1978 clearly links different positions on the rate of discount to different views of State role and powers.) -37- The OECD and UNIDO systems differed here too, not only on the meaning of optimum industrial development. For OECD "traded goods" was a prescriptive category: goods which should be traded in an optimum strategy. For UNIDO it was a predictive category: what in the analyst's best judgement, and subsequent to the use of his best arguments, will in fact be traded. There could be said to be a difference in political assumptions here: OECD assuming a Central Planning Office (CPO) able to dictate trading policy, UNIDO treating that as merely a special zero-constraint case (Sen 1972). LittleHirrlees (OECD) later accepted UNIDO's formal superiority here. But behind a miminal agreement on the relevance of what are the CPO's constraints (and inversely, its "control areas") there can be differences in political judgement: including different responses to uncertainty about constraints' location and malleability. OECD can be interpreted as judging that project planners must have simple guidelines, or that the prescriptive interpreta- tion of "traded goods" must be pressed forward in order to influence - even if only in the long run - opinions in government; UNIDO can be interpreted as the approach of planners who are more adaptive (or accessory), less combative (or presumptuous). The appendix illustrates this from multicomponent rural development projects: in considering later components, when should earlier components be considered sunk costs? the aggressive OECD evaluator with his eye on future - or past - battles may resist writing off costs; the adaptive UNIDO planner, focussed on the present decision, may not. One can undertake similar analyses for the remaining stages of assessment. This is done in subsections 3.4 to 3.6. The general reader may wish to instead proceed directly to section 4. 3.4 Generation and interpretation of data on demand and impact Having both considered direct costs and set the stage for analysing impacts, one will look at indicators of demand and use. Typical manoeuvres are: -38- (a) (b) (c) Use own data (Wilson 1973); avoid systematic experimental studies, e.g. as involving "unjust manipulation" of recipients. Argue that unfavourable assessments used unrepresentative sampling. Be optimistic on future prices and future demand indicators. In particular: - (d) (e) Adopt those demand growth indicators which are more favourable; quote past studies which assumed high growth, as a source of authority in themselves. Take past periods of high growth in demand for programme services as normal and representative; sideline periods of lower growth as "temporary" and "exceptional" (e.g. those due to international economic or domestic political crises); or: (f) Take past "trend" leged factors case. (3) 3.5 growth in demand as "normal"; add on for al- causing special additional generation in this Describing objectives and benefits To sell or save a programme, manoeuvres here include: (a) Propose and brandish splendid, broad, vague objectives. This is useful in both appraisal and evaluation.(4) "Goal language is ideological. Its function is to gain support" (Schaffer 1973, 14). Such objectives also help in cementing self-belief within organizations. (b) Particular stress the example is riding Vagueness is important for these roles. variants of (a) are to quote "national goals"; and to incomparable value of the particular objectives. An where access to remote areas is presented as an over- concern, a "right" regardless of cost and of the relative merits of an option of construction later out of surpluses from more productive present use of resources. Instead, in the "sacrifice mentality", costs part are benefits and costs are not comparable: of life, meeting them is merit-worthy, effort is -39- virtuous, benefits are benefits is sufficient (c) (d) (e) (f) (g) (h) (i) (j) therefore pure gain, and any level of as justification. Costs are not to be seen as foregone benefits elsewhere: all such calculation is immoral when "principles are at stake". In extreme variants constraints are not to be defined and respected but transformed by "doing better". Select favourable benefit measure methods. There are many opportunities other than the old favourite of double-counting benefits. Margolis 1973, Self 1975 and Stern 1976 give examples. (5) Claim important but presently undetailable net benefits (i.e. net of presently undetailable costs) - e.g. because fortune favours the bold. This is close to the "Columbus gambit", discussed earlier. It differs somewhat from Hirschman's "Hiding Hand" claim, that presently undetailable benefits may counterbalance presently detailable costs, so that if one forgets the former it is good to also forget some of the latter. Ex post, extend if necessary the scope of valued results, in a basic-needs programme have a very inclusive definition of basic needs. Also emphasize ex post whichever "incomparable" objective the programme did better on (e.g. environmental safeguarding, national security, public order and stability); almost any impact pattern can be defended in this way. Quote "testimonials" from individuals who received or saw benefi ts. Argue that subtle and general effects escaped identification. Stress the existence of longer-run benefits that have not yet been felt. Respecify objectives ex post, if that is more flattering; and/or impute criteria from outcomes, in self-justifying fashion. -40- 3.6 Synthesis of direct and indirect costs and benefits Finally one comes to the integration of cost and benefit assessment, to give an overall judgement. Some of the sales techniques are as follows: (a) Minimize rate of discount, because benefits tend to come later than costs. (Hanke & Yalker 1974 note this as a standard theme (b) (c) (d) and give illustrations). Be optimistic on the length of the project start-up period, i.e. minimize the lag of benefits behind costs, e.g. the time needed for execution of bureaucratic procedures). Choose the ranges of variation used in sensitivity analysis arbitrarily or according to one's convenience (Irvin 1978; Stewart 1978). York backwards from as yet unsatisfactory conclusions, revising elements so as to "fudge" up the assessment. "Fudge" down the assessment if the conclusion is too embarrassingly favourable. (e) Apply a "filter-in" approach: e.g. first approve programme elements on conventional economic (or even financial) grounds; then from the other elements approve some on social grounds; then some on "special grounds" such as inter-regional balance; etc. Do not filter-out, not apply criteria jointly (since this could rule out cases strong on just one ground). 4. PROJECT ASSESSMENT IN PERSPECTIVE 4.1 Questions Section 2 looked at the sociology of project assessment, and Section 3 at particular tactics. Ye should now pull together some of the biases that now appear possible, prior to the question of how far they could be countered, whether from within project analysis or only from outside it. The possible biases of insiders towards negatively, and of all parties towards neglect of -41- alternatives need no further emphasis; but three other possible biases should be mentioned. Firstly, the introduction of sophisticated formats for project appraisal and evaluation tends to favour those with the resources and skills to operate those formats. Secondly, some types of sector may and perhaps massage lend themselves better to those formats than do others, in terms of ability to generate data, claim a wide variety of benefits and so on. It has been suggested here that roads may be one such sector. Thirdly, suspected massage may lead assessors to impose a very high target rate of return (e.g. 10% or more). If that rate is used for discounting it may produce, apart from yet more vigorous massage, a bias against projects with long-run benefits. This is particularly significant in sectors where (unlike roads) projects can have very different net-benefit time profiles. (1) Further, if project analysis become devalued because people are aware of the likelihood of massage, this might further adhocery in their use; i.e. there would be less inhibition in rejecting projects which claimed high net returns or to retaining others despite poor results from analysis. One question is: what are the types and strengths of forces that may counter the motives and manipulations that have now been discussed? Yhat are the checks and balances? these might include: superior techniques of analysis; more effective organization of assessment and assessors, perhaps including use of independent outside consultants; and even tacit countertrends such as Hirschman's "Hiding Hand", whereby project salesmanship becomes seen as partly just healthy animal spirits. Subsection 4.2 touches on these factors. Another question is how this bargaining inside project analysis fits into the wider picture of political and administrative allocation; i.e. after looking and at the bargaining within project analyses, one can go on to ask how when the formats of project analysis are themselves fitted into overall -42- political bargaining subsection 4.3. Noting a variety various possible concerns their over of resources. possible forces Some perspectives are introduced in patterns of bargaining and allocation, and and counter-forces, the next question that arises relative importance. It is beyond the scope of the paper though to offer a predictive demography of relative frequencies of different patterns. There are in any case probably too many variables for that. The intention is only to underline the possibility, nature and significance of a of number kinds, comparable beyond the range of orthodox planning texts. This is to analysis of allocation via imperfect markets, which may high- light various possible bargaining patterns and constraining factors, without claiming to go administrative tions and on to predict net outcomes. allocation. tools that The the same seems to apply for hope then is simply to provide some ques- will help in facing the specifics of cases, and not encourage avoiding them by overgeneralizations. The aim of this concluding section is thus to sketch a set of issues worth bringing into project analysis training, not to attempt their detailed coverage or resolution. 4.2 Limited consolations Technique Bargaining and massage thrive on complexities. They are constrained if sensitivity analysis reveals that particular complexities are insignificant: e.g. that because this the further alternatives prospective benefits or further effects are not worth pursuing are less than the cost of search. (But is precisely the sort of issue that analysis often leaves unclear, and where judgements conflict.) -43- Even where complexities do matter, improved techniques might sometimes tame them: tighter experimental controls; goals and values rationalized into "logical frameworks"; and so on. Unfortunately such methods may commonly have excessive resource demands, especially for time: and new techniques and information channels sometimes for open self-interested as many disputes as they close, providing new ingenuity, while supervisory resources are always scarce. Critically, one is frequently in "wicked problem" situations (Rittel & Vebber 1973), marked by high complexity, uncertainty, dispute and flux. Much high-tech management seems thus to have proved of limited value even in DCs (see e.g. Rondinelli 1976). So concerning "logical frameworks", the difficulties with goals raised in 2.2 above are endemic and not just matters of technique. Donor-client relations may however have facilitated these methods' spread to international aid programmes; but they are poorly suited to activities with highly variable methods and hard-to-measure outputs (say, social as opposed to physical infrastructure), and to programmes with significant degrees of novelty, exploration, uncertainty, and broadness rather than narrowness of coverage - in other words to typical LDC conditions (Rondinelli 1983). The analyst negative ex various ways, will sometimes post findings repeated be helped by cumulation of findings. Vhile on one isolated rural road may be explained in negative findings from a variety of sources may generate a consensus (if not among non-economists) that rural roads are rarely worth considering in the absence of supporting programmes (Bovill 1978), notwithstanding proponents' ability to produce rosy appraisals apparently satisfying international norms. But cumulation is at best a longer-run check, and often absent when programmes and environments are changing rapidly. Improvements in analysis need not always be high-tech. For example, while checkers cannot counter thoroughgoing skilled fudging point-by-point, they can resort not only to sensitivity analyses to identify key assumptions, but also to spot-checks, and highlighting of leading alternatives. Their aim is -44- to ensure that different projects - presumed more or less equally fudged, in the absence of evidence to the contrary - are treated equally. The question returns though: why should checkers bother? This raises further issues for a sociology of assessment. Organization and socialization The orthodox ideal for what happens on the sort of stage we have considered is that each group has a distinctive role, and performs it in accord with a corresponding ethic. Such roles and ethics are often discussed for contractors or technical professionals and consultants; e.g. the latter are accorded a certain authority and privileges (including closed shop access to some types of work) in view of presumed conscientiousness and expertise. The issues need to be considered for others involved in project analysis too. In an optimistic view, misuse of methods will be reduced by more education, more effort and more supervision; "after the revolution" it may even be abolished. But for actors to follow such roles, certain conditions must hold. One is to have effective sanctions on performance. Apart from any other problems to wielding sanctions though, there are difficulties in determining what is unacceptable performance. Skilled work's quality cannot usually be judged by simple tests. Precisely because engineers et al are considered to be able to make complex judgements that outsiders cannot make, acquire a certain autonomy; though for social scientists and they evaluators, many others feel they can make the judgements as well if not better. To a significant degree, "if good work is to be done, it is necessary for the operatives to want to do it" (Ravetz 1973, 295). This requires internationalization of an ethic of conduct. Sustaining a group ethic and the associated morale in turn requires a solid group basis in distinctive skills, tolerable work conditions and rewards, and a group leadership with wider vision and sympathies. -45- These conditions do not hold automatically, as is seen for example from the familiar discussions of corruption among contractors (or state enterprises). Since contractors are not usually central actors in project appraisals and evaluations, different foci have been taken in this paper. Social scientists, for example, have in general won less professional status than have technical disciplines. Relatedly, if they were to act as "guardians of social efficiency", they would require resources, support and not least, mature policy sciences comparable to the mature technologies. Ye have considered cases where these conditions may not be met. (2) Specifically, fulfilment of the hope that consultants will provide disinterested independent expertise depends on the quality of client supervision and the state of the consultancy profession (see e.g. Hardiman & Midgley 1978). Commercial imperatives are powerful in the profession, and when - due to various factors supervision is deficient, shortcomings will appear. Consultants are a limited consolation within the assessment game: of value but, like any group, with interests of their own. Balances? If none of the individual checks imposed by technique and organization are very strong, they may still add up: "the facts", and cumulation of results over time; the professional ethics of practitioners and internal and external checkers; the occasional maverick "wistleblower". In addition there are the pressures of competition between different claimant and checker groups. The diagram below summarizes some of the tendencies and counter tendencies we have seen, comprising the full "assessment circus": assessment in the round. Countering possible organization interests in acceptance of bad programmes (2.2 above), outside assessors may become preoccupied with avoiding Type II errors. They may also be prone to a number of negative biases (2.4 above). In this light, practitioners' preoccupation with avoiding Type I errors appears more acceptable. -46ｾＭ Ｍ Witting or unwitting executive/ /practitioner interest .. Theories of Hiding Hand or organizational necessity First Law: commission of Type II errors (supporting bad programmes) - what outsiders fear • Outside assessors' preoccupation with avoiding Type II errors • I I· I I I I 1 . Practitioners' preoccupation with avoiding Type I errors . Role of supervision, control and rationing Second Law: commission of Type I errors (rejecting good prograDlJles) - what practitioners fear . Outsiders • biases or even hostility ----------------------------------------------------------------In some theories practitioners' typical initial overoptimism and subsequent defensiveness are even justified as necessary methods for getting programmes off the ground and then trying to survive through the inevitable learning period in the face of narrow critical assessments. (See e.g. Vildavsky 1979). One theory in this family is Hirschman's "Hiding Hand", where a bias in one direction (in this case, underestimation of some future difficulties) is counterbalanced by a bias in another (here, underestimation of one's ability to respond to the difficulties). One can, however, extend this analysis to obtain a fuller anatomy of biases (Gasper 1986) which does not give an expectation that optimistic cases (i.e. where biases cancel out) are likely to predominate. In fact, as Hirschman noted, even for the cases of balance, the learning effects underestimated ex ante can include not only positive new skills but also the skills of massage and fudge, which may then spread to other work. There is a danger of rationalizing almost many project history with these sorts of theory: as if any cost can be converted to a benefit if "experienced" in the proper way, and as if there were no better route to the benefits claimed. Hirschman's main suggestion in fact is to change the emphasis in, rather than the rigour of, ex ante appraisals: towards assessing adaptive potential, and away from -47- refinements of allocative calculus which are likely to be rendered irrelevant by the innumerable unpredictable factors in project situations. (See Gasper 1986 for an application to the roads sector, supporting the argument that ex ante overoptimism on roads projects is not excused by other considerations.) 4.3 More roles than one From a wider perspective on allocation one sees that the results of project analyses are certainly not necessarily the main determinants of decisions. Other struggles must be looked at besides those to claim a particular rate of return. Project analyses may often be: (a) omitted; or (b) ritualistic when undertaken (as sometimes seen in aid programmes, de facto commitments having been made at an earlier stage; Tendler, 1975); and/or (c) ignored as discussed in many reviews of non-utilization of studies (e.g. Rondinelli 1976, Veiss ed. 1977). These analyses emphasize that studies can figure in various roles in the bargaining of resources, not just in decisive roles. For a decisive role there must be developed assessment techniques, adequate study resources, clear and agreed decision-maker values, and issues which are politically "open". Study results can then have strong, direct and immediate influence. At another extreme, studies may be primarily a diversion: mystifying publics by their mere commissioning. Their results can still be used opportunistically, taken as sanction when in support of a desired conclusion, otherwise put into limbo. In the next role, studies of a particular type may be commissioned, because the general methods involved may tend to favour desired conclusions, even if not ensuring them. Some writers have claimed this for cost-benefit analysis, especially as it is seen in practice, with the textbook qualifications and refinements omitted or inevitably limited. A method which is not -48- inherently biased can yet be inevitably biased, depending on its effects in situations which are prevalent and will long remain so. To say that biases are "not inherent in the method of analysis, but only in the way in which it may be used by some practitioners" (A. Villiams 1972, 529) could be seriously misleading. Cost-benefit does however still have some universalist utilitarian thrusts, so it may often not serve particular interests; in which case its results may in many set ups only be used when convenient. systematic utilitarian-type formats do even so overall make it somewhat harder - compared to their absence and to reliance solely on other channels of struggle - to push through sectarian claims, and somewhat easier to push through welfarist ones; but they can also introduce new biases, and never remove the political element of "pushing through". If not just the general methods but also their particular. usage can be selected according to convenience, via massage, then study results can be manufactured as desired. Ve have seen that there are certainly limits and checks to massage, but they are fewer in the trademan's world than in the textbook, given the actual availabilities of information and of analytic, administrative and supervisory capacities. Less decisive roles for project analyses are as just one type of input to a wider political interaction, e.g. within "conflict management" and for "adversary" evaluation. They might even function sometimes as pure arbitration as political dice cum therapy; or instead at other times in unsettling but still non-decisive roles, intended to reduce complacency and stimulate review (Floden & Veiner 1978). Assessment techniques retain an invaluable role as frames encouraging collection of information and the putting of questions, such as on opportunity costs and indirect effects. Even if in practice the answers quite often are obscure, the methods do produce valuable constraints on allocation simply according to power. -49- 4.4 Perspectives for research and training One interesting area for research concerns the actual degrees of robustness/manipulability of current project analysis methods and assessments. The issue of the scope and limits to manipulation needs to be pursued, as part of the theory of administrative allocation. Relatedly, attention should be given to the extent of variation between contexts and sectors. It is felt that the themes raised in the paper have a wide relevance: the conflicts between insiders and outsiders, and doers and deliberators, the bargaining for administered resources, and the scope for judgements and massage in the associated project analyses. Similarly, it is not felt that the argument applies only to the roads sector, which provided most of the examples. But further work is needed to refine the analysis, by qualification according to type of economic sector, macro-political/economic environment, micro-institutional setting (e.g. which particular agents are involved in bargaining), and the level of issue being bargained about. (3) These sorts of analyses should then feed through into training. Project analysis training has to cover much more than these matters; but the emphasis here has been on rectifying an existing deficiency. One must look not just at intellectually challenging techniques but also the contexts and content of their actual use. This discussion will often (certainly in this paper) be less formally rigorous than the imposing techniques. It is less neatly disciplinary, and also more embarrassing! But it is facing relevant questions and experiences. These should not be left in the shadows, tacitly recognized as important but not examined at some stage in training. Training should provide an awareness of the roles of analysis in the real world of constraints, interests and politics, where planning is limited - not just produced - by problems. Planning theory and have characteristically been discordant. Put simply, the positive as well as the normative/prescriptive Positive planning theory must be one of the inputs planning experience training must cover theory of planning. to the prescriptive -50- theory. Otherwise irrelevance. training in project planning is in danger of academic -51- APPENDIX: BASE CASES AND SUNK COSTS IN RURAL ROAD PROJECTS Integrated rural development programmes usually contain road components, sometimes as a sister project rather than within the main programme's administration. Assessment of such road projects ideally would like to compare cases 1 and 2 below, and then for different with-road cases estimate the optimal roads provision within the total programme. Yith road Case 1 Case 3 Yithout road Case 2 Yith rural development (RD) programme Case 4 Yithout rural development programme But estimation of case 2 may be very difficult. Case 4 may be easier to grasp, with fewer counter-factual steps distancing it from reality. In other words, we typically treat the road and the RD elements as a joint project, and compare case 1 (with both) and case 4 (without both). The economist will at once start to worry about the exclusion of good proposals through their combination with bad, or vice versa. Assessing joint projects of various compositions is of course the answer. Also easy to answer in theory are presentations of components of the total programme which to some degree claim joint benefits (compare 1 and 4) but only attribute their own costs (compare 1 & 3, or 1 & 2). The implicit or explicit claim is that the other programme components are sunk costs. The easy theoretical answer is that if those costs are indeed sunk then they should be ignored now; if not then they should be accounted. But there may be some complications. For example, in one District in Malawi two recent studies assess joint programmes of road and other rural development expenditures. The RD project document cost feeder roads, but does not specify or cost the new trunk road on which its estimates of induced agricultural benefits appear conditional. Instead it claims all those benefits for itself. Perhaps the trunk road -52- will be constructed impact. in But perhaps not. any case, regardless of its local agricultural The trunk road project document certainly did not feel it could afford to ignore that impact and claimed some of the projected induced benefits projects are for itself. concessionally A contributory foreign factor here is that these financed and the documents are very probably presented to different donors. Consider the detailed planning of a roads project intended to complement an ongoing rural development programme in a certain area. was always envisaged but priority in this roads component earlier stages was given to other components, for which major expenditures have already been made, with others planned. Vhich essaying one of these non-road expenditures should be costed when now a detailed design and justification of the roads? would consider that certainly prospective At first glance levels of expenditure in excess of levels in areas not covered by these special IRD programmes should be costed. These are resources potentially usable elsewhere. it is that, felt effect while committed cause of already new not yet expended, these resources are already in in political statements and/or loan agreements, or be- local expended But what if levels of expectation? And what of resources in fact - e.g. in building up project staffing, offices, housing, etc - but done so in the expectation of having their impact jointly with the eventual roads component? that expenditure happened. and In principle a joint project was assessed before found attractive. In practice this may even have But with the elapse of time a new assessment is being made before the roads funds are committed, to check on any unforseen circumstances. latest off phase all tional of one Integrated Rural Development Programme in Malawi wrote past "routine"l"recurrent" costs, though this covers much institudevelopment, previously justified research, by training, etc. directed reference to the later phase. to and as given: now project IRR was 35%. routine unconditional perhaps It also attributed none of these activities' ongoing costs to a 'new-phase project'. taken The commitments. They were The estimated -53- The first case - unspent "commitments" - is like the OECD-UNIDO division on traded goods. The OECD style evaluator will say that an alternative presently being ignored is still an alternative, and these opportunity costs must be weighed. The UNIDO style evaluator may reply that there is no opportunity for influence in this case; to which the other may say that he will not enter into speculations on influence: his role - either in general qua evaluator, or in this case as formally briefed - is to assess in terms of policy-makers' stated general objectives. The UNIDO evaluator in contrast may accept or feel unable to resist decision makers' specific actions. The second case - spent commitments - appears clearer: the costs are clearly sunk. But should a district where the roads programme is scheduled later be treated differently from one where because not all districts could be taken at once it was scheduled earlier? The implication might be to do two assessments for the later district: one costing all commitments, to review the total conception, and one ignoring sunk costs, to guide the current decision. But suppose the review of total conception and past commitments shows them to be unfortunate; that this appears due to lack of adequate (or any) initial appraisal and the later components only payoff on the assumption of sunk past costs? For the UNIDO style evaluator the current decision is unaffected; he must simply try to better control future appraisals. The OECD style evaluator may ask if that control can be furthered not merely by publicizing the past "error", but also by seeking to penalize it: making an object lesson, refusing to support a follow-up investment if it only "piggybacks" on an unjustified (according to the evaluator) past commitment. Yhich leads us back to the arguments about possible influence and appropriate roles. The implicit reference in this discussion was to various mUlti-component programmes in Malawi, the role of the administratively and financially separate District Roads Improvement project, and to sunk costs of a particular small-holder cash-crop agency. Similar issues arise within multiroad programmes; if road 1 was built on the basis of a claim which proves to -54- have been invalid (e.g. a particular level of diverted traffic), should it fearlessly be proclaimed a sunk cost and road 2 be undertaken in addition (e.g. upgrading of the road from which traffic did not divert)? The question is what lay behind the invalid claim. I have heard road engineers in this context discuss with fascination the fearlessness, yet also possibly naivete, of the economist pursuing rational allocation. There can be fallacies of disjointed incrementalism and of "letting byegones be byegones". -55- NOTES TO SECTION 1 1. Initial comparisons of conventional road cost-benefit analyses with the outputs of more general planning models are however seriously unfavourable to the former (Hine 1982, 22). 2. See Phillips 1977, also Patton 1978, papers in Abt (ed) 1977, and Downs & Rocke 1981 on disagreements between evaluations. 3. On formal equivalence of the OECD, UNIDO and IBRD systems, see e.g. Squire & van der Tak 1975. The comparisons of the OECD and UNIDO systems of project evaluation took place within the new awareness from the 60's of how past project choice had neglected opportunities in world trade, leading to low and even negative net value added in some LDC industries. But the common opinion, that the UNIDO system was a clumsy simplified version of OECD, sometimes neglected the most important issues at stake: whether optimum industrial development and maximum international trade involvement coincide. The OECD manual is part of a wider Vestern official view, stressing "dynamic benefits" from trade and the development of strengths though facing world competition. UNIDO is closer to an UNCTAD line, e.g. a concern with negative externalities in production and consumption which markets neglect, including openness to foreign pressure, moulding of consumer tastes, etc. Vithout pursuing this debate the point to note is that the differences between the wider judgements are more important than the (associated) differences in explicit method. 4. Laver (1978, 473): "Some game theorists have developed ... analysis of the extensive options often available within the rules of the game. Almost never do these take into account ... the informal alternatives of, e.g. sophistication, bluffing, lying and cheating. It is my contention that this type of informal strategy is much more of the essence of realpolitik than the extensive analysis •.• [produced by] the formal models industry". NOTES TO SECTION 2 1. Caiden and Vildavsky (1974, 110) note similar switches, between the different stages of domestic budgetary negotiations. And some supposed "checkers" avoid conflict, give up, "join the club". 2. "Vhat to the evaluator is nothing more than reporting the facts (or "the mildest constructive criticism") can be seen by the manager as anything from malicious sabotage to vicious assault (Gurel 1975, 20). 3. Rondinelli argues that elaborate consultancy studies have shown little impact on resulting project effectiveness (1976, 599). But one question is what the studies' terms-of-reference told them to look at. -56- 4. For example, exclusion of leisure-time-savings benefits in most LDC studies; or minimal analysis of changing vehicle composition, or of vehicle lives and depreciation, even though these may dominate operating costs in some cases. Steve Mandel has shown me the great sensitivity of results in one case to assumptions on this last issue, e.g. the common assumption of a given vehicle lifetime kilometrage, independent even of road surface type (vehicle maintenance supposedly repairs all surface-related wear). But this assumption could bias assessment against bitumenization, i.e. it is not a lobbyist's ruse. 5. More attention is given here to roads than buildings, since the latter figure less in economic planning theory and practice: they are less susceptible to analysis of benefits, they are usually smaller expenditures and they are less involved in the rituals of foreign donor finance. Architects also emphasize that the building is often a very small part of lifetime system costs, including personnel and other operating costs, so that priority is to a building which facilitates good working. But: there is no necessary relation between high cost and functionality; while there is almost unlimited scope for varying designs and increasing costs; and the question of function itself must not be begged, as in many projects which assumed that great numbers of new offices, houses, etc. were necessary. Expenditure on buildings usually adds up to an extremely high share of the government budget, and economists must consider it. But the methods used mostly have to come from outside their cost-benefit manuals. One key issue is choosing which precedent-derived norms on level of facilities should be fought for, when the pruner is hard pressed to raise support for worrying over 'trifling' infractions. This is the standard issue of where to expend energies combating 'drift'. 6. No programmes are found to produce the desired effect, at least when studies " •.• gather data independently of the agency; adopt a short time frame that either minimizes the chance for the desired effect to appear, or if it does appear permits one to argue that the results are temporary and probably due to [the Hawthorns effect]; and maximize the search for other variables that might explain the effects observed" (James Q. Vilson 1973, 133). 7. Bauer (1975) goes further, suggesting a link between the prevalence of 1960's and 70's low or negative value added LDC aided industrial projects, and oversold aid project analyses. I suspect the claim may be more applicable to the far greater outpouring of 1970's commercial bank loans, with bankers winning promotion elsewhere with each new superficially analyzed deal? NOTES TO SECTION 3 1. Hanke and Valker 1974 took a sample of 128 projects, of which more than 75 % showed cost overruns. the group as a whole exceeded original -57- estimated cost by 72 %. Rondinelli reviewed highway projects in Central America under 4 different agencies: "the behaviour of all 19 highway projects is quite similar. That is they all experienced similar patterns of delay and unexpected cost increases" (p. 579». 2. This may seem gross. Indeed there is little theoretically complex to use here but it is still a good practical ploy. Consider analysis of a road network with traffic proceeding down numerous routes. It may be necessary to define a simplified base case to make assessment calculations manageable. Choice of a flattering base case does not discriminate between new route alternatives, if they are all compared with it. All are equally flattered. This may be discounted in some other part of the assessment but it may not. Definition of base case traffic routing is a 'detail', not to be highlighted in a report. 3. Consider projections of normal traffic growth in road appraisal studies. Available indicators, such as GDP growth, vehicle licensing figures, fuel imports/production, manual traffic counts, and automatic traffic counters, are all notoriously imperfect guides. They also offer a wide spread of suggestions. So there is scope for interpretation and selection (e.g. of imports of one grade of fuel that have risen fast - even though another grade has been tailing off). Studies also make mutually endorsing use of each other's claims, even though earlier claims may be outdated or unfulfilled or for non-comparable road stretches. Normal traffic growth can be exaggerated in at least two ways: (i) interpretation of 'normie', i.e. 'how it should be' so that only trends from good spells are taken; (ii) identification with past total growth, even though that growth included new generation from new roads, etc., which cannot be double counted when assessing the impact of future new roads. Note that normal traffic gets twice the weight of generated traffic. Road engineers are usually conservative in design, and may sometimes then feel that they should countervailingly be favourable to the proposal in traffic analysis. (Design conservatism is to avoid the professional stigma of premature product collapse). but one cannot correctly balance now-committed construction costs by exaggerated benefit claims. A more significant countering point is that generated traffic does appear in quite a few cases to have been under estimated. 4. However in so far as a project was oversold at appraisal by listing such objectives, this may later generate compensating evasions in monitoring and evaluation if those objectives are still being referred to (Deutscher 1976). 5. Consider also the following: Value the time savings from preventing road closures in the rainy reason by taking the salaries of potential users for the prevented time of closure (Final Report - not Draft Final; i.e. accepted by donor agency as well as client Government - on Secondary Roads in an African country). In effect, assume that users -58- sit and wait on the road, and do not divert to other routes or other work! Again this enters as 'detail', not highlighted in a report. NOTES TO SECTION 4 1. Less distorting funds. would be use of a 2. For example, morale raised through a form of combative group-spirit may not further analysts' effectiveness, and/or can soon wither. A number of "leadership" tasks can then be suggested as potentially valuable but undoubtedly difficult. 3. Analysis will tend to be more decisive for specific-issue and single programme analyses than at strategic and mUlti-programme levels (Carley 1980, 30), and for small changes in well understood environments. ｕｎｉｄｏＭｳｾｹｬ･＠ premium on investment -59- REFERENCES ABT, C.C., Hills. ed., 1976, The Evaluation of Social Programs; Sage, Beverly Alexander, E., 1982, Design in the decision-making process; Policy sciences, 14, 3, 279-92. Arrow, K., 1976, in C.C. Abt (ed.) 1976. Bauer, P.T., 1975, N.H. Economics, 2,4, 387-405. Stern on substance and method; J. of Development Boulding, K., (eds.), 82-92. Economist 1964, The and the Engineer; in Smith & Castle Bovill, D., 1978, Rural road appraisal methods for developing countries; U.K. Transport and Road Research Laboratory. Branch, T. 1971, Courage without esteem; Vashington Monthly, 3,3, 23-40. Brewer, G., 1981, Vhere the Twain Meet: Reconciling Science and Politics in Analysis; Policy Sciences, 13,3, 269-79. Bridger, G., 182, Rapid Project Appraisal; Crown Agents, London. Broad, V. & N. Vade, Science; O.U.P., London. 1985, Betrayers of the Truth: Fraud and Deceit in Caiden, N., & Vildavsky, A., 1974, Planning and Budgeting in Poor Countries; 1980 edition, Transaction Books, New Brunswick, NJ. Carley, M., 1980, Rational Methods in Policy Analysis; Heinemann, London. Carruthers, I., 1977, Applied 0.0.1. Review, 1977/2, 12-28. project appraisal: the state of the art; Cline, M., 1976, The 'what' without the 'why'; in C.C. Abt (ed.), 1976, 36774. Deutscher, l., 1976, Towards avoiding the goals trap in evaluation research; in C.C. Abt (ed.) 1976, 249-68. Downs, G. & Rocket D., 1981, Complexity, interaction and policy research; Policy Sciences, 13,3, 281-95. Edmonds, G., & Howe, J., (eds.), 1980, Roads and Resources; lTDG, London. Fitzgerald, E.V.K., 1978, Public Sector Investment Planning for Developing Countries; Macmillan, London. -60- Floden, R., & Yeiner, S., 1978, Rationality to Ritual; Policy Sciences, 9,1, 9-18. Gasper, D., 1986, The Hiding Hand and other stories; forthcoming. Gurel, L., 1975, The human side of human services programs; in M. Gittentag & E. Struening (eds.), Vol. 2, 11-28. Guttentag, M. & Struening, E., Research; Sage, Beverly Hills. (eds.), 1975, The Handbook of Evaluation Hanke, S. & Yalker, R., 1974, Benefit-cost analysis reconsidered; in Haveman & Margolis (eds.), 329-54. Hardiman, M. & Midgley, J., 1978, Foreign consultants and development projects; J. of Administration Overseas, XVII, 4,232-44. Haveman, R. & Margolis, J., (eds.), 1977, Public Expenditure and Policy Analysis; 2nd edn., Rand McNally, Chicago. Heyer, J. et al Macmillan, London. (eds.), 1981, Rural development in Tropical Africa; Hicks, N., 1982, Internal Yorld Bank paper on aid evaluation. Hill, P., 1982, Synthesis of Rural Road Evaluations; ODA; London. J., 1982, Road Planning for Rural Development Hine, Countries; U.K. Transport and Road Research Laboratory. Hirschman, A., Yashington. in Development 1967, Development Projects Observed; Brookings Institution, Hodgkinson, C., 1978, Blackwell, Oxford. Towards a Philosophy of Administration; Basil I.D.A., 1982, IDA in Retrospect; Yorld Bank, Yashington. Irvin, G., 1975, Roads and Redistribution; ILO, Geneva. Irvin, G., 1978, Modern Cost-Benefit Methods; Macmillan, London. Janis, I., 1972, Victims of Groupthinki Houghton Miflin, Boston. Johnson, K., 1981, Book review; Third Yorld Planning Review, 3,1, 98-9. Joy, L., 1973, Remarks at Yorkshop on Research Priorities in the Economics of Agricultural Development; DEV, University of East Anglia. Joy, L. n.d., Yhat an economist wants to know about dams; ODA, London. -61- Lappe, F., et aI, 1980, Aid as Obstacle; Institute for Food and Development Policy, San Francisco. Laver, M., 1978, How to be sophisticated, lie, cheat, bluff and win at politics; Political Studies, XXVI, 4, 462-73. Layard, R. 1972, Penguin, London. Introduction to R. Layard (ed.), Cost-Benefit Analysis; Lindblom, C., 1968, The Policy Process; Prentice-Hall, Englewood-Cliffs, NJ. Linsley, R., 1964, Engineering and economics in project planning; in Smith & Castle (eds.), 93-103. Margolis, J., 1973, Technical Advice on Policy Issues; Sage, Beverly Hills. Morgan, P., 1983, Social analysis, project development and advocacy in U.S. foreign assistance; Public Administration and Development, 3,1, 61-71. The O.D.A., 1983, Administration, London. Lessons O.D.M., 1978, A Guide to Developing Countries; U.K. Devt. Min., HMSO, London. of Experience; Overseas Development Transport Planning in the Roads Sector for Transport and Road Research Lab., for Overseas Patton, M., 1978, Utilization-Focused Evaluation; Sage, Beverly Hills. Phillips, 59. D.C., 1977, Vhen evaluators disagree; Policy Sciences, 8,2, 147- Ravetz, J., 1973, Scientific Knowledge and its Social Problems; Peregrine edn., Penguin, London. Rein, M. & Schon, D., 1977, Problem setting in policy research; in C. Veiss (ed.), 1977, 235-51. Rittel, H. & Vebber, M. 1973, Dilemmas in a general theory of planning; Policy Sciences, 4,2, 155-69. Rondinelli, D., 1976, International assistance policy and development project administration; International Organization, 30,4, 573-605. Rondinelli, London. D., 1983, Development Projects as Policy Experiments; Methuen, Sampson, A., 1981, The Money Lenders; Hodder & Stoughton, London. Schaffer, B., 1973, Policy Discussion paper, IDS Sussex. Decisions and Institutional Evaluation; IDS -62- Scriven, M., 1971, Goal-free evaluation; in d. Hamilton et al (eds.), 1977, Beyond the Numbers Game, Macmillan, London. Self, P. 1975, Econocrats and the Policy Process; Macmillan, London. Sen, A.K., 1970, Interrelationships between project, sector and aggregate planning; Econ. Bulletin for Asia & the Far East, 1/2, 66-75. Sen, A.K., 486-501. 1972, Control areas and accounting prices; Econ. journal, 82, Simon, H. & A. Stedry, 1969, Psychology and economics, in G. Lindley & E. Aronson (eds.), The Handbook of Social Psychology; Reading, USA. Smith, S. & Castle, E., (eds.), 1964, Economics and Public Policy in Water Resources Development; Iowa State U.P., Ames, Iowa. Smith, T.B., 1985, Evaluating development policies and programmes in the Third World; Public Admin. and Devt., 5,2, 129-44. Squire, H. & Van der Tak, H., 1975, Economic Analysis of Projects; Johns Hopkins, U.P., Baltimore. Stern, G., 1976, SOSIPing; Operational Research Quarterly, 27,4, 915-29. Stewart, F., 1978, SCBA in practice; World Development, 6, 153-65. Suchman, E., 1967, Evaluative Research; Russell Sage Foundation, New York. Tendler, J., 1975, Inside Foreign Aid; Johns Hopkins U.P., Baltimore. Thornton, D.S., 1973, Remarks at Workshops on ... (as in Joy 1973 above). Weiss, C., (ed.), 1977, Heath, Lexington, Mass. Using Social Research in Public Policy Making; Wildavsky, A., 1979, The Art and Craft of Policy Analysis: Speaking Truth to Power; Little, Brown & Co., Boston. Wilensky, H. & C. Lebeaux, 1958, Industrial Society and Social Welfare; Russell Sage Foundation, New York. Wiles, P., 1977, Economic Institutions Compared; Basil Blackwell, Oxford. Williams, A., 1972, Cost-benefit analysis: bastard science? andlor insidious poison in the body politick?; in Haveman & Margolis (eds.), 519-45. Williamson, O.E., 1982, Book review; J. of Economic Literature, XX,2, 585-6. -63- Wilson, F., 10,1, 24-6. Wilson, J .0., Interest, 30. 1978, 1973, Ex-post On evaluation pettigrew and and sectoral policy; IDS Bulletin, armor: an afterthought; Public