See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/10884808

Outcomes research: What is it and why does it matter?

Article in Internal Medicine Journal · April 2003

DOI: 10.1046/j.1445-5994.2003.00302.x · Source: PubMed

CITATIONS READS

43 9,998

3 authors:

Michael Jefford Martin Stockler

Peter MacCallum Cancer Centre The University of Sydney
323 PUBLICATIONS 7,463 CITATIONS 576 PUBLICATIONS 15,108 CITATIONS

SEE PROFILE SEE PROFILE

Martin H N Tattersall
The University of Sydney
645 PUBLICATIONS 32,083 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Hepatocellular (HCC) clinical Carcinoma trials View project

Preferences for immunotherapy in Melanoma View project

All content following this page was uploaded by Michael Jefford on 06 December 2017.

The user has requested enhancement of the downloaded file.

Internal Medicine Journal 2003; 33: 110–118

REVIEW

Outcomes research: what is it and why does it matter?

M. JEFFORD, M. R. STOCKLER and M. H. N. TATTERSALL
1Ludwig Institute for Cancer Research, Heidelberg, Victoria, 2Department of Medicine, Department of Public Health and
Community Medicine and the National Health and Medical Research Council Clinical Trials Centre, and 3Department
of Cancer Medicine, University of Sydney, New South Wales, Australia

Abstract present review details the historical background of

outcomes research to reveal the origins of its diver-
Outcomes research is a broad umbrella term without sity. The value and relevance of outcomes research,
a consistent definition. However it tends to describe commonly employed research techniques and exam-
research that is concerned with the effectiveness of ples of recent publications in the area are also dis-
public-health interventions and health services; that cussed. (Intern Med J 2003; 33: 110–118)
is, the outcomes of these services. Attention is fre-
quently focused on the affected individual – with
measures such as quality of life and preferences – but Key words: cost-benefit analysis, decision support
outcomes research may also refer to effectiveness of techniques, health-care quality, access and evaluation,
health-care delivery, with measures such as cost- outcome and process assessment (health care), quality
effectiveness, health status and disease burden. The of life.

WHAT IS ‘OUTCOMES RESEARCH’? Outcome assessment (health care) is defined as:

There appears to be no consistent definition of what research aimed at assessing the quality and
constitutes health outcomes research. It has been effectiveness of health care as measured by the
suggested that ‘outcomes research’ is a difficult term attainment of a specified end result or outcome.
to define because of the breadth of research included Measures include parameters such as improved
under this ‘umbrella expression’.1 The US National health, lowered morbidity or mortality, and
Library of Medicine does not include the term as a improvement of abnormal states (such as elevated
medical subject heading (MESH).2 Related terms blood pressure).
include ‘health services research’ and ‘outcome
assessment (health care)’. Health services research is The term ‘outcomes research’ describes a variety of
defined as: fields of research that use a variety of methodologies,
often with differing aims.1 The US Agency for
the integration of epidemiologic, sociological,
Healthcare Research and Quality suggests that:
economic, and other analytic sciences in the study of
health services. Health services research is usually
outcomes research seeks to understand the end
concerned with relationships between need, demand,
results of particular health care practices and
supply, use, and outcome of health services. The aim
interventions. End results include effects that people
of the research is evaluation, particularly in terms of
experience and care about, such as change in the
structure, process, output and outcome.2
ability to function. In particular, for individuals with
chronic conditions – where cure is not always
possible – end results include quality of life as well as
Correspondence to: Michael Jefford, Peter MacCallum Cancer Institute, mortality. By linking the care that people get to the
Locked Bag 1, A’Beckett Street, Vic. 8006, Australia. outcomes they experience, outcomes research has
Email: Michael.Jefford@petermac.org become the key to developing better ways to monitor
Received 24 October 2001; accepted 10 April 2002. and improve the quality of care.3
 What is outcomes research? 111

An assessment of the historical factors that have did not explain variations in the use of services. Other
shaped the field is required in order to understand the studies around this time, including further obser-
diversity of outcomes research. vations by Wennberg et al.,13,14 reported variation in
the use of hospital services yet noted that health
HISTORICAL BACKGROUND outcomes were not compromised. These studies
emphasized the need for research to determine the
In 1913, a Massachusetts surgeon, Ernst Codman, causes and implications of differences in practice.
noted that hospitals reported the number of patients
treated, but did not indicate whether patients Around this time McNeil et al. reported a study that
appeared to benefit from treatment.4 He argued that was designed to assess preferences for treatment,
hospitals should produce a report of their treatment involving a trade between improved quality and
results, standardized to allow comparison between quantity of life, following the diagnosis of laryngeal
hospitals. Furthermore, he suggested that all aspects cancer.15 The study surveyed the attitudes of healthy
of hospital practice should be evaluated to ensure that volunteers and found that, to maintain their voices,
they produced favourable outcomes. This includes approximately 20% of volunteers would chose radi-
not only a focus on patient encounters, but also an ation treatment over surgery. The study was important
analysis of all ‘products’ of the hospital including, for in that it emphasized the use of outcomes that were
example, the efficiency of administrative procedures patient-focused.
and staff and student training.
In the 1988 Shattuck lecture, Ellwood called for the
The term ‘outcome’ was coined by Donabedian, who development of ‘outcomes management, a “tech-
developed a paradigm for quality assessment, nology of patient experience”’.16 Reviewing the
comprising structure, process and outcome.5 He preceding changes to the US health system, Ellwood
recognized that, while some outcomes (such as death) stated that, ‘the most destabilizing consequence of the
might be easily recognized and measured, others restructuring of the health system has also been, in my
might be less easily defined and measured. Among view, the most desirable one: patients, payers, and
the latter he included ‘patient attitudes and satisfac- executives of health care organizations now have both
tions, social restoration and physical disability and higher expectations and greater power’. The problem,
rehabilitation’.6 Donabedian suggested that, he claimed, was that:
‘outcomes, by and large, remain the ultimate valida-
too often, payers, physicians, and health care
tors of the effectiveness and quality of medical care’.
executives do not share common insights into the life
He has continued to guide the assessment of quality.7
of the patient…the problem is our inability to
measure and understand the effect of the choices of
In 1973, Wennberg and Gittelsohn reported wide
patients, payers and physicians on the patient’s
variation in medical practice within Vermont, USA.8
aspirations for a better quality of life. The result is
There was considerable variation in resource input,
that we have uninformed patients, skeptical payers,
use and expenditure among neighbouring commun-
frustrated physicians, and besieged health care
ities but without obvious adverse sequelae. Wennberg
executives.16
et al. attributed this variation to differences in phys-
icians’ diagnostic style and to physicians’ beliefs in the He called for a technology of ‘outcomes manage-
efficacy of particular treatments.9,10 In 1982, ment’, designed to allow all parties to make rational
McPherson et al. confirmed variations in the use of choices based on improved insights into the effects
six common surgical procedures in the USA, United that those choices have upon the lives of patients.
Kingdom and Norway.11 There appeared to be a lack Outcomes management, he discussed, involves:
of consensus regarding effective medical procedures (i) increased reliance on standards and guidelines,
and techniques. There also seemed to be no clear link (ii) the collection of clinical outcome data (as well as
between these procedures and favourable results. patient-completed functional and well-being data),
Chassin et al. queried whether inappropriate use of (iii) pooling and analysis of data through the use of
procedures could explain geographical variation of centralized databases and (iv) subsequent dissemin-
results.12 They studied the use of coronary angi- ation of results and modification of guidelines.
ography, carotid endarterectomy and upper-
gastrointestinal-tract endoscopy in a variety of Based upon the recognition of wide variations in prac-
regions. Although they recognized a high incidence of tice, evidence of inappropriate use and the need for
inappropriate use, Chassin et al. concluded that this health-care evaluation, the US Health Care Financing

Internal Medicine Journal 2003; 33: 110–118

112 Jefford et al.

Administration and the Public Health Service began The framework developed by Lee et al. allows us to
to assume an active role in providing information to recognize both overlapping and exclusive elements
guide medical practice.17 of these fields at the levels of: (i) research topics,
(ii) outcomes, (iii) secondary analysis and (iv) appli-
Reflecting upon Ellwood’s vision, Relman suggested cations. Clinical trials are predominantly designed to
that the era of assessment and accountability repre- assess the efficacy and effectiveness of specific inter-
sented the third revolution in medical care, following ventions. The goal of outcomes research may include
the ‘era of expansion’ of health services from the the assessment of quality of care, access and the effec-
1940s through the 1960s and the ‘era of cost contain- tiveness of general health-care strategies. The typical
ment’ beginning in the 1980s.18 outcomes in clinical trials research are survival,
response and adverse event rates. The outcomes in
Paralleling these changes was an evolving shift from a
typical outcomes research studies are processes, costs
provider-centred model of medicine to patient-
and health-related quality of life. Methods of
centred health care.19 Patient-based outcomes and
secondary analysis, such as meta-analysis and
greater accountability therefore became more relevant
decision analysis, may use the same data sets, but are
and important.
frequently considered to be elements of clinical and
Available governmental funding further drove outcomes research, respectively. Clinical trials
outcomes research in the USA. Initial projects funded research is aimed primarily at informing clinical deci-
by the Agency for Health Care Policy and Research sions, whereas outcomes research is aimed primarily
were determined on the basis of ‘the number of at informing health policy decisions.
individuals affected, the extent of uncertainty or
controversy with respect to the use of a procedure or However, outcomes research and clinical trials
its effectiveness, the level of related expenditure, and research have not developed in isolation. Outcomes
the availability of data’.20 Initially, prospective, rand- research has progressed beyond studies of effective-
omized studies were discouraged. Projects funded ness to encompass quality of care, decision analysis
subsequently have encouraged the generation of and the analysis of administrative databases.
primary data regarding the effectiveness of interven- Modern clinical trials often include measures of
tions, either through randomized trials or prospective health status, quality of life, resource utilization and
longitudinal studies. Thus outcomes research has cost. Lee et al. recognize that the distinctions
come to describe a range of activities (Table 1). between clinical trials research and outcomes
research are useful conceptually, but may be over-
Lee et al. have proposed a conceptual framework of simplifications in practice.
how outcomes research interfaces with clinical trials.1
Thus, clinical trials and outcomes research are
different, but they complement, rather than compete
Table 1 The scope of outcomes research with, each other. Pronovost and Kazandijan provide
some innovative examples of how the concepts of
Outcomes research may focus on:
clinical trial design and outcomes research can be
Quality-of-life measures
Effectiveness integrated to answer questions about quality improve-
Cost ment using modifications of case-control studies and
Quality of care randomized trials.21
Patient preferences
Appropriateness THE VALUE OF OUTCOMES
Access RESEARCH
Health status
In areas such as: Outcomes research complements clinical trial
Disease prevention research. It aims to: (i) provide better information to
Screening inform patient decisions, (ii) guide health providers
Drug treatment and (iii) inform health policy decisions. In Table 2 we
Medical procedures have suggested ways that outcomes research might
Medical practices benefit the individual patient, the health-care prac-
Diagnostic tests
titioner, the health-care organization and the
Guidelines
government. These benefits may also be derived from
Health-care policy
clinical research and other means.

Internal Medicine Journal 2003; 33: 110–118

 What is outcomes research? 113

Table 2 Suggested benefits of outcomes research

Consumer
Increased participation in decision-making
Increased choice regarding hospital/practitioner/treatment options
Assurance regarding effectiveness of interventions
Assessment and development of interventions to improve well-being, not just survival
Health-care provider
Greater certainty regarding the benefit of an intervention
Standards/guidelines to guide clinical practice
Shared responsibility in decision-making
Protection from malpractice suits (if complying with above)
Health-care organization management
Greater use of effective interventions
Discontinuation of ineffective interventions/practices
An organizational culture emphasizing quality
Cost savings as inappropriate use is eliminated (i.e. interventions, medications, hospitalizations)
Government
Cost savings as inappropriate use is eliminated (i.e. interventions, medications, hospitalizations)
Greater ability to plan health services
Only effective pharmaceuticals and services are subsidized
Target research in areas of greatest potential impact based on examination of databases etc.

EXAMPLES OF RECENT combined with chemotherapy). The patient charac-

PUBLICATIONS IN OUTCOMES teristic most highly correlated with receipt of
RESEARCH radiation was age at diagnosis.

Below we briefly review four recent publications to The data suggest that adjuvant radiation is under-
illustrate aspects of outcomes research. used. This information would not be found with
traditional clinical trials research. This study also
Who gets adjuvant treatment for stage II and
identified important areas for further research, for
stage III rectal cancer?
example, the exploration of physicians’ and patients’
Method: the use of linked databases knowledge and attitudes about adjuvant radiation
Value: greater use of effective interventions therapy.
Schrag et al. conducted a retrospective cohort study
Cost-effectiveness of radiofrequency ablation
to examine the relationship between patient charac- (RFA) for supraventricular tachycardia (SVT).
teristics and the use of adjuvant pelvic radiation with
or without chemotherapy.22 American patients with Method: decision analysis
stage-II or stage-III rectal adenocarcinoma were iden-
Value: cost savings, greater use of effective interventions
tified from the linked Surveillance, Epidemiology and
End-Results, and Medicare databases. Analysis of Cheng et al. used a decision-analysis model to
coding identified episodes of surgery consistent with compare the health and economic outcomes of three
definitive tumour resection that occurred ≤6 months treatment strategies for patients with SVT: (i) initial
after diagnosis. Medicare coding identified the appro- RFA, (ii) long-term antiarrhythmic drug therapy or
priate procedural and revenue codes consistent with (iii) treatment of acute episodes of arrhythmia with
the use of radiotherapy and chemotherapy. Informa- antiarrhythmic drugs.23 Costs were estimated from a
tion was also collected on: (i) type of surgical major academic centre and from the literature, and
procedure, (ii) comorbidities and (iii) demographic treatment effectiveness was estimated from reports
data (i.e. age, gender, race and median income for of clinical studies. Probabilities of clinical outcomes
geographical area). The study found that only 57% of were also based on published data. Utility data were
patients received adjuvant radiation (mostly based upon patient-reported quality of life before

Internal Medicine Journal 2003; 33: 110–118

114 Jefford et al.

and after RFA. This study concluded that RFA Randomized trial comparing traditional Chinese
substantially improves quality of life and reduces medical acupuncture, therapeutic massage, and
expenditures when used to treat symptomatic self-care education for chronic low back pain
patients. RFA improved quality-adjusted life expect- Method: overlap between the traditional clinical trial and
ancy by 3.1 quality-adjusted life years (QALY) and outcomes research
reduced lifetime medical expenditure by $US27 900,
compared with long-term drug therapy. Long-term Value: assessment and development of interventions to
drug therapy was more effective and had lower costs increase well-being and increased certainty regarding the
compared with episodic drug treatment. The benefit of interventions
findings were highly robust in sensitivity analyses; The study of Cherkin et al. randomly allocated
that is, similar results were obtained when multiple patients with chronic lower-back pain to traditional
factors were varied across a wide range of plausible Chinese medical acupuncture, therapeutic massage or
values. self-care education.26 The primary outcomes were
symptoms and dysfunction. Secondary outcomes
This study included a measure of patient preference included disability, health-care use and cost. Patients
(utility assessment) and economic analysis and were assessed 4, 10 and 52 weeks after a 10-week
reported measures of marginal effectiveness. Decision intervention period. At baseline and at each follow-up
analysis provides a rational approach for decision visit patients were asked how ‘bothersome’ back pain,
making in areas where definitive data are lacking, and leg pain, numbness and tingling had been in the
particularly in identifying areas of critical uncertainty. preceding week, each on a scale from ‘0’ to ‘10’.
Completeness of safety reporting in randomized Patients also completed the Roland Disability Scale.
trials Automated health-care use data were collected,
including a record of all provider visits, medications
Method: survey
dispensed, imaging procedures, operations and hospit-
Value: emphasis on quality at a procedural/organizational alizations. With respect to primary end-points, the
level study found that therapeutic massage was effective for
persistent lower-back pain and appeared to confer
Ioannidis and Lau recently addressed the quality of
long-lasting benefit, as assessed by 1-year follow-up.
safety reporting in clinical trials.24 They surveyed
192 randomized studies within seven topics of This study exemplifies the interface between tradi-
internal medicine and assessed the adequacy of tional clinical trials and outcomes research. The study
reporting of adverse effects. Principal measures used patient-focused outcome data, including
were: (i) the severity of adverse clinical events and measures of health perception (symptom panel) and
laboratory abnormalities and (ii) the frequency and measures of function (disability scale), in a rand-
reasons for withdrawals due to toxic effects. The omized clinical trial. Quality of life is the most
study found that the quality and quantity of safety important outcome in a clinical scenario such as this.
reporting varied, but was generally inadequate. The The study also illustrates the application of adminis-
severity of clinical adverse events and laboratory- trative data to explore important unconventional
determined toxicity was adequately defined in only outcome measures in the rigorous context of a rand-
39% and 29% of trial reports, respectively. Only omized trial.
46% of trials stated the frequency of specific reasons
for discontinuation of study treatment due to SOME TECHNIQUES/
toxicity. The authors concluded that the standard of METHODOLOGIES IN OUTCOMES
collection, analysis and reporting of safety data in RESEARCH
clinical trials lags behind that for efficacy data. They
suggested that the outcome measures used in this Because of the breadth of outcomes research, a dis-
study could be incorporated into the recommend- cussion of all approaches and methodologies is
ations of the Consolidated Standards of Reporting impossible. However several approaches are com-
Trials statement.25 monly used, and these are briefly discussed below.

This study demonstrates the diversity of important Measures used to assess well-being and
outcomes and measures of quality. Measures of satisfaction
quality may be applied to areas beyond the direct Clancy and Eisenberg have provided a useful descrip-
patient–practitioner focus. tion of the broad aspects of health-related quality

Internal Medicine Journal 2003; 33: 110–118

 What is outcomes research? 115

of life.27 Health perceptions may be assessed through be useful for comparing therapies with similar
symptom panels. Several validated symptom outcomes, it is not possible to compare interventions
measures have entered routine clinical practice. For with different outcomes using this method. In addi-
example the American Urological Association tion, it can be difficult to take into account more than
symptom index scale is used by >80% of practising one component of a health strategy, such as the
urologists in the USA.28 Functional measures may be adverse effects of drug therapy.
used to assess the impact of health interventions on a
range of domains, including physical, mental, social Cost-utility analyses account for the possibility that
and role function. Preference-based measures convey the not all outcomes are valued equally by weighting
meaning that a person places upon their individual outcomes according to their perceived value (utility).
health status. Patient satisfaction may also be directly Years in good health are likely to be valued more than
assessed. This may include interpersonal and tech- years with sickness. For example, gains in survival
nical aspects of care. may be in varying states of imperfect health, due to
the effects of the disease or treatment. To account for
Instruments may be generic or disease-specific.29 this, the life-years gained in a particular state may be
Generic instruments allow comparison between multiplied by the utility of that health-state to give
different conditions and between different interven- QALY. Utility refers to the value attributed to a
tions. They may also detect the differential effects particular health-state (outcome). Utility measures
that an intervention has on different aspects of health are an example of a generic measure of health-related
status. Specific instruments focus on: (i) single quality of life because they provide a summary of
disease states (such as asthma), (ii) patient groups quality of life that can be compared across diseases,
(such as the frail elderly), (iii) areas of function (such conditions and populations.
as sleep) or (iv) a particular problem (such as pain).29
Specific instruments are often considered to be more Cost-utility analyses are expressed in terms of dollars
clinically sensible. They may also be more sensitive to per QALY gained. For example, the incremental cost
changes in specific aspects over time. The validity of in the USA of adjuvant chemotherapy for node-
many scales has been established. Acknowledging the positive breast cancer in pre-menopausal women has
tension between ‘generalizeability’ and specificity, been estimated to be about $US10 000 per QALY
several groups have recommended a modular gained.36
approach, which incorporates instruments that recon-
cile these conflicting needs by adding specific items to Decision analysis
a generic core.30–33
Sarasin describes decision analysis as the ‘quantitative
Economic analysis application of probability and utility theory to
decision making under conditions of uncertainty’.37
Economic analysis includes techniques such as Decision analysis involves modelling a problem to
cost outcome, cost-effectiveness and cost-utility
guide decision-making. The model builds on
analyses.34
available clinical information such as: (i) prevalence
Cost-outcome studies are descriptive and indicate the of disease, (ii) effectiveness of interventions,
costs associated with a particular disease or treatment (iii) incidence of side-effects, (iv) associated costs and
strategy. The lack of information about efficacy and (v) outcome measures, such as patient-assigned
the lack of a suitable comparison often limit the utility values. Thus, decision analysis is able to link
usefulness of such studies. For example, the median research results, patient preferences and population
cost in Canada of treating women with advanced data. Sensitivity analysis involves varying the assump-
ovarian cancer with second-line and subsequent tions made within the model. For example, the risk of
chemotherapy was estimated to be approximately side-effects may be higher (or lower), or the effective-
$Can37 000; however, costs and outcomes in the ness of an intervention may be lower (or higher) than
absence of such treatment were not available.35 in the baseline model. Similarly, patient preferences
(reflected in utility values) may vary. Sensitivity
Cost-effectiveness studies compare the incremental cost analysis allows one or many factors to be adjusted,
of one treatment over another with the incremental alone or simultaneously, to determine whether this
benefit in terms of a single common outcome.34 For affects the apparent best choice; that is, how sensitive
treatments that influence survival, the cost can be the decision is to the assumptions on which the model
expressed in terms of life-years gained. While this can is based.

Internal Medicine Journal 2003; 33: 110–118

116 Jefford et al.

Decision analysis can be particularly useful where mation. They are normally centralized and, with
there is inadequate information available from clinical mandatory reporting, offer the potential for thorough,
studies to guide treatment or policy recommen- accurate reporting. One disadvantage is the frequent
dations. Its greatest benefit here is to identify key lack of a denominator or reference population.
areas of uncertainty requiring further research.
Kassirer et al. also suggest that decision analysis can Clinical trial databases
be used: (i) to address health policy questions about These contain detailed clinical information regarding
screening and prevention, (ii) to weigh trade-offs trial participants. Because of this, detailed and multi-
between tests and treatments and (iii) for the inter- variate analysis within the data set may be possible.
pretation of clinical data where there is uncertainty.38 Disadvantages include the potential difficulty in
generalizing to a broader population and the time and
Data sources in outcomes research expense required to collect high-quality data.
The use of databases facilitates data exploration
through the analysis of interventions and outcomes Censuses
and, thus, through the generation of novel insights. A Censuses provide information on the demographic
wide variety of health-information sources exists, composition of large populations with respect to such
including administrative databases, clinical databases, variables as age, gender, ethnicity, place of residence,
disease registries and clinical-trial databases. These employment and education. Importantly, these data
datasets may be linked with information from other may be used as a denominator for other data (such as
sources, such as census or electoral role data. Issues cancer incidence) to determine rates. However,
surrounding databases and outcomes research were collecting census data is expensive, logistically diffi-
discussed at the Regenstrief Conference Measuring cult and limited to the collection of a relatively small
Quality, Outcomes, and Cost of Care Using Large Data- amount of information.
bases and reported in a supplement to Annals of
Internal Medicine in 1997.39 Linking datasets may overcome the individual limit-
ations of the various data sources. Different
Administrative databases information sources may be combined for modelling
in decision analysis. Inferences may also be made
Administrative data are obtained from health-care
when different information sources are juxtaposed,
administration, member enrolment and the collection
which may serve to generate hypotheses or prompt
of payments for services.40 Producers of such data
further research.
include federal and state governments and private
health insurers. Clinical content is often low and may PUBLIC REPORTING OF MEDICAL
only include demographic details and coding infor- OUTCOMES
mation concerning diagnoses and procedures.
Nevertheless, administrative data are important At first glance, the public reporting of health out-
because they are readily available, inexpensive to comes seems an important social responsibility. In
acquire, computer-readable and, typically, cover large the Bristol case in the United Kingdom, it is possible
populations. that more public presentation of outcomes might
have avoided deaths.42 The New York State Depart-
Clinical databases ment of Health collects data on quality of care pro-
Clinical databases may include laboratory, radiology, vided to patients undergoing coronary artery bypass
pharmacy and surgery scheduling systems. They grafting.43 The publication of surgeon-specific death
contain detailed clinical information, covering a rates raised many concerns. Many physicians lack the
broad population. However, a major disadvantage of skill to interpret and critically evaluate medical liter-
these datasets is that the information is often recorded ature,44 which leads us to ask, Can consumers be
in a non-standardized manner and is difficult to expected to appraise outcomes data? Concerns such
access. Thus, it is often necessary to manually extract as independent data validation and appropriate risk
data elements. This is expensive, time-consuming and adjustment may not be adequately considered. Public
prone to error.41 reporting may lead to hospitals or physicians avoiding
high-risk patients in an attempt to lower risk-adjusted
Disease registers mortality rates. Regarding the public release of infor-
Disease registers, such as cancer registers, provide mation on organizational performance, Epstein sug-
incidence data and, frequently, detailed clinical infor- gests that attention be given to determining what

Internal Medicine Journal 2003; 33: 110–118

 What is outcomes research? 117

constitutes a valid assessment of quality and how to by applying a scientific approach to the evaluation of
ensure the integrity of reported data.45 He states that, a wider range of clinical, management and organi-
‘it has become clear that providing information to zational problems.
consumers in a way that is understandable and allows
them actually to use it is at least as formidable a task REFERENCES
as developing reliable and valid quality indicators’.45
1 Lee SJ, Earle CC, Weeks JC. Outcomes research in oncology:
history, conceptual framework, and trends in the literature.
THE AUSTRALIAN SITUATION J Natl Cancer Inst 2000; 92: 195–204.
Australia, like other developed countries, is conf- 2 National Library of Medicine. Medical Subject Headings
[cited 2002 September 18]. Available from: URL:
ronted with the challenge of providing an equitable, http://www.nlm.nih.gov/mesh/
efficient, affordable and high-quality health system. 3 Agency for Healthcare Research and Quality. Outcomes
Cost containment is necessary. The shift to casemix- Research Fact Sheet. AHRQ Publication no. 00-P011.
based funding has produced gains in throughput and Rockville: Agency for Healthcare Research and Quality; 2000.
has reduced costs.46 Casemix classification systems Available from: URL: http://www.ahrq.gov/clinic/outfact.htm
4 Codman EA. The product of a hospital. Surg Gynecol Obstet
have also encouraged inter-hospital comparisons on 1914; 18: 491–6.
issues such as the planning of bed and staff numbers, 5 Donabedian A. Evaluating the quality of medical care.
utilization review and funding allocations. What is Milbank Mem Fund Q 1966; 44: 166–206.
missing from casemix is an assessment of quality of 6 Kelman HR, Willner A. Problems in measurement and
care (including the process) and cogent measures of evaluation of rehabilitation. Arch Phys Med Rehab 1962; 43:
172–81.
patient outcomes. 7 Donabedian A. The quality of care. How can it be assessed?
JAMA 1988; 260: 1743–8.
In Australia, the quality of health-care delivery is 8 Wennberg JE, Gittelsohn A. Small area variations in health
partly regulated by legislation (i.e. compulsory regis- care delivery. Science 1973; 182: 1102–8.
tration of health-care providers) and also regulated by 9 Wennberg JE, Gittlesohn A. Variations in medical care among
the requirement for accreditation of health-care facil- small areas. Sci Am 1982; 246: 120–34.
ities. Australia has embraced the importance of 10 Wennberg JE, Barnes BA, Zubkoff M. Professional uncertainty
and the problem of supplier-induced demand. Soc Sci Med
alternative outcome measures in the process of 1982; 16: 811–24.
approval of drugs to the Pharmaceutical Benefits 11 McPherson K, Wennberg JE, Hovind OB, Clifford P. Small-
Scheme (PBS). All applications include a cost-effec- area variations in the use of common surgical procedures. an
tiveness analysis. The standard of health-related international comparison of New England, England, and
economic evaluations in Australia was questioned by Norway. N Engl J Med 1982; 307: 1310–4.
12 Chassin MR, Kosecoff J, Park RE, Winslow CM, Kahn KL,
Salkeld et al. in 1995.47 Although Hill et al. suggest Merrick NJ et al. Does inappropriate use explain geographic
that there are significant problems with many variations in the use of health care services? A study of three
pharmacoeconomic analyses, they report that the procedures. JAMA 1987; 258: 2533–7.
intensive evaluation process used in the PBS allows 13 Wennberg JE, Freeman JL, Culp WJ. Are hospital services
for identification and correction of many problems.48 rationed in New Haven or over-utilised in Boston? Lancet
1987; 1: 1185–9.
14 Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital
Although outcomes research in Australia has not been
use and mortality among Medicare beneficiaries in Boston and
funded as generously as in the USA, Australian New Haven. N Engl J Med 1989; 321: 1168–73.
outcomes research and resulting publications are 15 McNeil BJ, Weichselbaum R, Pauker SG. Speech and survival:
increasing. Australian clinical trialists have pioneered tradeoffs between quality and quantity of life in laryngeal
and led the incorporation of patient-based measures cancer. N Engl J Med 1981; 305: 982–7.
16 Ellwood PM. Shattuck lecture – outcomes management.
and economic analyses into clinical trials.49,50
A technology of patient experience. N Engl J Med 1988; 318:
1549–56.
CONCLUSIONS 17 Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H.
Effectiveness in health care. An initiative to evaluate and
Outcomes research aims to provide information to all improve medical practice. N Engl J Med 1988; 319:
stakeholders (including patients, health-care 1197–202.
providers, health-care managers and government) to 18 Relman AS. Assessment and accountability: the third
allow more rational, evidence-based decision-making. revolution in medical care. N Engl J Med 1988; 319: 1220–2.
Outcomes research complements clinical trials and 19 Laine C, Davidoff F. Patient-centered medicine. A
professional evolution. JAMA 1996; 275: 152–6.
other more traditional forms of clinical research. 20 Agency for Health Care Policy and Research. Medical
Outcomes research also offers the prospect of Treatment Effectiveness Research. Rockville: Agency for
improving the quality of other aspects of health care Health Care Policy and Research; 1990.

Internal Medicine Journal 2003; 33: 110–118

 118 Jefford et al.

21 Pronovost PJ, Kazandjian VA. A new learning environment: 35 Doyle C, Stockler M, Pintilie M, Panesar P, Warde P,
combining clinical research with quality improvement. J Eval Sturgeon J et al. Resource implications of palliative
Clin Pract 1999; 5: 33–40. chemotherapy for ovarian cancer. J Clin Oncol 1997; 15:
22 Schrag D, Gelfand SE, Bach PB, Guillem J, Minsky BD, 1000–7.
Begg CB. Who gets adjuvant treatment for stage II and III 36 Hillner BE, Smith TJ. Efficacy and cost effectiveness of
rectal cancer? Insight from surveillance, epidemiology, and end adjuvant chemotherapy in women with node-negative breast
results – Medicare. J Clin Oncol 2001; 19: 3712–8. cancer. A decision-analysis model. N Engl J Med 1991; 324:
23 Cheng CH, Sanders GD, Hlatky MA, Heidenreich P, 160–8.
McDonald KM, Lee BK et al. Cost-effectiveness of 37 Sarasin FP. Decision analysis and its application in clinical
radiofrequency ablation for supraventricular tachycardia. Ann medicine. Eur J Obstet Gynecol Reprod Biol 2001; 94: 172–9.
Intern Med 2000; 133: 864–76. 38 Kassirer JP, Moskowitz AJ, Lau J, Pauker SG. Decision analysis:
24 Ioannidis JP, Lau J. Completeness of safety reporting in a progress report. Ann Intern Med 1987; 106: 275–91.
randomized trials. an evaluation of 7 medical areas. JAMA 39 Anonymous. Measuring quality, outcomes, and cost of care
2001; 285: 437–43. using large databases. Proceedings of the 6th Regenstrief
25 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I et al. Conference. Marshall, Indiana, 4–6 September 1996. Ann
Improving the quality of reporting of randomized controlled Intern Med 1997; 127: 665–774.
trials: the CONSORT Statement. Jama 1996; 276: 637–9. 40 Iezzoni LI. Assessing quality using administrative data. Ann
26 Cherkin DC, Eisenberg D, Sherman KJ, Barlow W, Intern Med 1997; 127: 666–74.
Kaptchuk TJ, Street J et al. Randomized trial comparing 41 McDonald CJ, Overhage JM, Dexter P, Takesue BY, Dwyer
traditional Chinese medical acupuncture, therapeutic massage, DM. A framework for capturing clinical data sets from
and self-care education for chronic low back pain. Arch Intern computerized sources. Ann Intern Med 1997; 127: 675–82.
Med 2001; 161: 1081–8. 42 Bolsin SN. Professional misconduct: the Bristol case. Med J
27 Clancy CM, Eisenberg JM. Outcomes research: measuring the Aust 1998; 169: 369–72.
end results of health care. Science 1998; 282: 245–6. 43 Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards
28 Barry MJ, Fowler FJ Jr, O’Leary MP, Bruskewitz RC, of reporting medical outcomes publicly. N Engl J Med 1996;
Holtgrewe HL, Mebust WK. Measuring disease-specific health 334: 394–8.
status in men with benign prostatic hyperplasia. Measurement 44 Anonymous. How to read clinical journals: II: to learn about a
Committee of the American Urological Association. Med Care diagnostic test. Can Med Assoc J 1981; 124: 703–10.
1995; 33: AS145–55. 45 Epstein AM. Rolling down the runway: the challenges ahead
29 Guyatt GH, Feeny DH, Patrick DL. Measuring health-related for quality report cards. JAMA 1998; 279: 1691–6.
quality of life. Ann Intern Med 1993; 118: 622–9. 46 Health Solutions Pty Ltd. Independent Assessment of
30 McHorney CA. Generic health measurement: past Casemix in Victoria. Melbourne: Health Solutions Pty Ltd;
accomplishments and a measurement paradigm for the 21st 1994.
century. Ann Intern Med 1997; 127: 743–50. 47 Salkeld G, Davey P, Arnolda G. A critical review of health-
31 Cella DF, Tulsky DS, Gray G, Sarafian B, Linn E, Bonomi A related economic evaluations in Australia: implications for
et al. The Functional Assessment of Cancer Therapy Scale: health policy. Health Policy 1995; 31: 111–25.
development and validation of the general measure. J Clin 48 Hill S, Henry D, Mitchell A. Problems in pharmacoeconomic
Oncol 1993; 11: 570–9. analyses. JAMA 2000; 284: 1922–4.
32 Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, 49 Coates A, Gebski V, Bishop JF, Jeal PN, Woods RL, Snyder R
Duez NJ et al. The European Organization for Research and et al. Improving the quality of life during chemotherapy for
Treatment of Cancer QLQ-C30: a quality-of-life instrument advanced breast cancer. A comparison of intermittent and
for use in international clinical trials in oncology. J Natl Cancer continuous treatment strategies. N Engl J Med 1987; 317:
Inst 1993; 85: 365–76. 1490–5.
33 Aaronson NK, Bullinger M, Ahmedzai S. A modular approach 50 Stewart RA, Sharples KJ, North FM, Menkes DB, Baker J,
to quality-of-life assessment in cancer clinical trials. Recent Simes J. Long-term assessment of psychological well-being in a
Results Cancer Res 1988; 111: 231–49. randomized placebo-controlled trial of cholesterol reduction
34 Detsky AS, Naglie IG. A clinician’s guide to cost-effectiveness with pravastatin. The LIPID Study Investigators. Arch Intern
analysis. Ann Intern Med 1990; 113: 147–54. Med 2000; 160: 3144–52.

Internal Medicine Journal 2003; 33: 110–118

View publication stats