Evaluation of

influenza vaccine effectiveness

A guide to the design and interpretation of

observational studies

Immunization, Vaccines and Biologicals

 Evaluation of influenza vaccine effectiveness: a guide to the design and
interpretation of observational studies

ISBN 978-92-4-151212-1

© World Health Organization 2017

Some rights reserved. This work is available under the Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO;
https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this licence, you may copy, redistribute and adapt the work for non-
commercial purposes, provided the work is appropriately cited, as indicated below. In
any use of this work, there should be no suggestion that WHO endorses any specific
organization, products or services. The use of the WHO logo is not permitted. If you
adapt the work, then you must license your work under the same or equivalent Creative
Commons licence. If you create a translation of this work, you should add the following
disclaimer along with the suggested citation: “This translation was not created by the World
Health Organization (WHO). WHO is not responsible for the content or accuracy of this
translation. The original English edition shall be the binding and authentic edition”.

Any mediation relating to disputes arising under the licence shall be conducted in
accordance with the mediation rules of the World Intellectual Property Organization.

Suggested citation. Evaluation of influenza vaccine effectiveness: a guide to the design and
interpretation of observational studies. Geneva: World Health Organization; 2017.
Licence: CC BY-NC-SA 3.0 IGO.
Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.

Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/

bookorders. To submit requests for commercial use and queries on rights and licensing, see
http://www.who.int/about/licensing.

Third-party materials. If you wish to reuse material from this work that is attributed to a
third party, such as tables, figures or images, it is your responsibility to determine whether
permission is needed for that reuse and to obtain permission from the copyright holder.
The risk of claims resulting from infringement of any third-party-owned component in the
work rests solely with the user.

General disclaimers. The designations employed and the presentation of the material in this
publication do not imply the expression of any opinion whatsoever on the part of WHO
concerning the legal status of any country, territory, city or area or of its authorities, or
concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps
represent approximate border lines for which there may not yet be full agreement.
The mention of specific companies or of certain manufacturers’ products does not imply
that they are endorsed or recommended by WHO in preference to others of a similar nature
that are not mentioned. Errors and omissions excepted, the names of proprietary products
are distinguished by initial capital letters.

All reasonable precautions have been taken by WHO to verify the information
contained in this publication. However, the published material is being distributed without
warranty of any kind, either expressed or implied. The responsibility for the interpretation
and use of the material lies with the reader. In no event shall WHO be liable
for damages arising from its use.

Printed in Geneva, Switzerland

 Evaluation of
influenza vaccine effectiveness

A guide to the design and interpretation of

observational studies

Immunization, Vaccines and Biologicals

 Contents
Acknowledgements ii
Abbreviations & acronyms iii

1. Introduction 1
1.1 Purpose of this guide and target audience 1
1.2 Epidemiology of influenza 2
1.3 Influenza vaccines 3
1.4 Organization of this guide 4

2. The role of VE studies in influenza vaccination programmes 5

2.1 Challenges to studying the impact of influenza
vaccination programmes 6
2.2 U se of VE studies to estimate impact of influenza
vaccination programmes 7

3. Outcomes of interest for influenza VE studies 8

4. Assessing influenza vaccination history 15
5. Measuring covariates 18
5.1 Key covariates 19
5.2 Other covariates 20
6. Study designs 22
6.1 Cohort studies 22
6.2 Case-control studies 24
6.3 T
 he test-negative design: a special instance of
case-control evaluation 25
6.4 The screening method 26

7. Statistical considerations 27
7.1 Sample size 27
7.2 Characterizing the study participants 28
7.3 Analytic approaches to estimate crude rate/odds ratios 28
7.4 Assessing potential confounders 29
7.5 Final analyses 29
7.6 Pooling data from multiple VE studies 29
8. Building a test-negative study onto SARI surveillance 31
8.1 SARI case definition 32
8.2 Laboratory testing 32
8.3 Timing of enrolment 33
8.4 Other considerations 33

9. Reporting influenza VE results 34

9.1 Description of influenza vaccines 34
9.2 Description of circulating influenza viruses 35
9.3 Stratification of VE estimates 35
References 36
Annex 1. Glossary of terms 46
 Acknowledgements

The preparation of this publication was coordinated by the Initiative for Vaccine
Research of the WHO Department of Immunization, Vaccines, and Biologicals.

The lead writer was Michael L. Jackson of the Group Health Research Institute,
Seattle, WA, USA. The document was initially discussed in a WHO expert consultation
in February 2014. The following participants at this consultation provided document
development advice and technical input during the writing process: Alicia M. Fry
(Centers for Disease Control and Prevention, Atlanta, GA, USA), Mark A. Katz
(Ben Gurion University of the Negev, Beer-Sheva, Israel), Carlos Lara (University
of Antwerp, Antwerp, Belgium), Danuta M. Skowronski (British Columbia Centre
for Disease Control, Vancouver, Canada), Dipika Sur (Consultant THSTI, Delhi,
India), John C. Victor (PATH, Seattle, WA, USA), and Syed M.A. Zaman (Medical
Research Council Unit, The Gambia).

Others provided helpful reviews and suggestions during the public comment period:
ACOG Immunization Expert Work Group (American College of Obstetricians and
Gynecologists, Washington, DC, USA), Jon S. Abramson (Wake Forest School
of Medicine, Winston-Salem, NC, USA), Ben Cowling (The University of Hong
Kong, Hong Kong SAR, China), C. El Guerche Seblain (Sanofi Pasteur, Lyon,
France), Nathalie El Omeiri (Pan American Health Organization/World Health
Organization, Washington DC, USA), Jill M. Ferdinands (Centers for Disease
Control and Prevention, Atlanta, GA, USA), Chuanxi Fu (Guangzhou Center
for Disease Control and Prevention, Guangzhou, China), Alan R. Hinman (Task
Force for Global Health, Decatur, GA, USA), Terri B. Hyde (Centers for Disease
Control and Prevention, Atlanta, GA, USA), Peter B. McIntyre (National Centre
for Immunisation Research and Surveillance of Vaccine Preventable Diseases, The
Children’s Hospital at Westmead and the University of Sydney, New South Wales,
Australia), Jozef J. P. Nauta (Abbott EPD, Weesp, Netherlands), Wilfred Ndifon
(African Institute for Mathematical Sciences, Cape Town, South Africa), S. I. Samson
(Sanofi Pasteur, Lyon, France), James A. Southern (Advisor to Medicines Regulator,
Simon’s Town, South Africa), and Sheena G. Sullivan (WHO Collaborating Centre
for Reference and Research on Influenza, Melbourne, Australia).

Technical coordination and input was provided by Justin R. Ortiz of the WHO
Initiative for Vaccine Research. Additional technical review for WHO was provided
by Thomas Cherian, Adam L. Cohen, Joachim Hombach, and Ole Wichmann.

WHO thanks the US Centers for Disease Control and Prevention for providing
funds to the WHO Initiative for Vaccine Research which supported the production
of this document.

ii
 Abbreviations & acronyms

ACIP Advisory Committee on Immunization Practices

of the United States of America

ARI Acute respiratory infection

CDC Centers for Disease Control and Prevention

of the United States of America

COPD Chronic obstructive pulmonary disease

EPI Expanded Programme on Immunization

FluCAN Influenza Complications Alert Network in Australia

GISRS WHO Global Influenza Surveillance and Response System

HA Influenza hemagglutinin glycoprotein

Hib Haemophilus influenzae type b

IIV Inactivated influenza vaccine

ILI Influenza-like illness

I-MOVE Influenza Monitoring of Vaccine Effectiveness (I-MOVE)

network in Europe

ICU Intensive care unit

IVR WHO Initiative for Vaccine Research

NA Influenza neuraminidase glycoprotein

PCV Pneumococcal conjugate vaccine

RCT Randomized clinical trial

REVELAC-I Multicentre network for evaluation of influenza vaccine

effectiveness in Latin America

RT-PCR Reverse transcription-polymerase chain reaction

SARI Severe acute respiratory infection

SPSN Canadian Sentinel Practitioner Surveillance Network

STROBE Strengthening the Reporting of Observational Studies

in Epidemiology

VE Vaccine effectiveness

WHO World Health Organization

iii
 1. Introduction

1.1 Purpose of this guide and target audience

Influenza causes an estimated 444 000–553 000 deaths annually worldwide.1 Most of
these deaths occur among elderly adults and persons with chronic cardiopulmonary
disease, but the toll includes 28 000–111 000 deaths among children less than five
years of age. 2 In view of this global disease burden, many countries have implemented
influenza vaccination programmes in order to reduce morbidity and mortality due
to influenza. WHO recommends that countries consider influenza vaccination for
persons at high risk of death or serious illness from influenza and has recognized
the following groups to be at increased risk: pregnant women, children aged 6–59
months, elderly adults, individuals with specific chronic medical conditions, and
health-care workers.3

Historically, influenza immunization programmes have been implemented mainly

in high-income countries. In recent years, however, middle-income countries
have initiated influenza immunization programmes, and policy-makers in low-
and middle-income countries are increasingly assessing whether and how to
implement new influenza immunization programmes. Policy-makers have faced
similar decisions regarding introduction of other vaccines in recent years, including
vaccines against rotavirus,4 Haemophilus influenzae type b (Hib), and Streptococcus
pneumoniae.5 Decisions about introducing new vaccines are based on a thorough
understanding of the burden of disease, the performance of the vaccines in preventing
severe disease, the anticipated impact of a programme, national capacity, and other
appropriate considerations. Efforts to expand introduction of rotavirus, Hib, and
S. pneumoniae vaccines have led to well-established guidance for evaluating the
effectiveness of vaccines and impact of vaccination programmes.6 However, several
features of influenza epidemiology and vaccines create unique challenges for the
evaluation of vaccine effectiveness (VE) and the expected benefits of influenza
vaccination programmes. In this guide these challenges are described and the role
of observational (non-randomized) influenza VE studies in evaluating influenza
vaccination programmes is discussed. The designs for such studies are described and
their advantages and limitations reviewed.

This guide is written primarily for researchers who design observational influenza
VE studies and for public health scientists who interpret and apply the results of these
studies. For researchers, it illustrates critical considerations in the design and analysis
of influenza VE studies, as biased results may be produced even in settings where
data completeness and quality are high. For public health scientists, it shows why VE
studies have had a larger role in policy for influenza vaccines than for other vaccines.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 1
The guide also addresses the critical evaluation of influenza VE studies by describing
study limitations and errors that can lead to biased VE estimates and the contextual
information that is necessary to properly interpret VE results.

Importantly, WHO does not recommend that VE studies be conducted by all countries
with influenza vaccination programmes. VE studies are being conducted by a number
of networks worldwide.7-9 The results of these existing studies can be expected to be
applicable in other countries which have similar influenza epidemiology and similar
vaccination programmes, in terms of vaccines used and risk groups targeted. The
decision to carry out VE studies should be based on a need for country-specific VE
estimates and on capacity to conduct rigorous VE studies.

1.2 Epidemiology of influenza

Influenza disease is primarily caused by influenza A and B viruses, which are spread
from person to person by respiratory droplets and fomites.10 In temperate regions,
both influenza A and B cause winter epidemics, with sporadic cases and outbreaks
occurring out of season. In tropical regions, circulation of influenza viruses is more
complex, ranging from seasonal peaks similar to those in temperate regions, to two or
three annual incidence peaks, or to year-round virus circulation.11

The influenza virus genes that encode the viral surface proteins undergo rapid
genetic changes during viral replication. These accumulating genetic changes lead to
changes in the conformation of surface proteins, a process known as antigenic drift.10
Antigenic drift results in reduced acquired immunity over time as antigenically
novel viruses replace influenza viruses that previously circulated in the population.
Influenza A viruses may also undergo a more dramatic change known as antigenic
shift. In antigenic shift, virus strains emerge with novel surface antigens to which
population immunity is low or absent. This is typically the result of reassortment
of animal and human influenza A viruses. Such viruses have the potential to cause
global pandemics.12 Cumulative incidence of influenza virus infection during seasonal
epidemics may reach 20–30% in children13 and 5–10% in adults14 and it has been
estimated that influenza causes 444 000–553 000 deaths per year globally.1 The risk
of severe influenza may differ between high- and low-resource settings. 2

Individuals at increased risk for complications of influenza virus infection include

children <5 years of age, pregnant women, elderly adults, and individuals with
chronic health conditions such as chronic heart or lung disease, asthma, and HIV/
AIDS. Elderly adults have the highest rates of influenza-associated hospitalizations
and deaths of any age group and may account for up to 90% of influenza-associated
deaths from seasonal influenza viruses. Children aged <5 years, and particularly
children aged <2 years, also have high rates of influenza-associated complications,
with an estimated 1–2 million cases of influenza-associated severe acute lower
respiratory infections and 28 000–111 500 deaths annually. 2 Pregnant women also
have an increased risk of severe illness due to influenza.15

2 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
1.3 Influenza vaccines
Current trivalent influenza vaccines contain antigens of influenza A (H1N1), influenza
A (H3N2) and one influenza B strain. Quadrivalent vaccines contain antigens of two
influenza A strains and two B strains (Victoria and Yamagata lineages). The antigenic
composition of the vaccines is adjusted each year. In an effort to match the antigens
in the vaccine to those of the strains expected to circulate in the subsequent winter,
the antigenic composition of the vaccines is reviewed prior to production for each
hemisphere, with choice of formulation based on the characteristics of circulating
influenza viruses tested within the WHO Global Influenza Surveillance and Response
System (GISRS).16 Currently, influenza vaccines are produced at two separate times
of the year, once for the northern hemisphere (with distribution generally beginning
in September) and once for the southern hemisphere (with distribution generally
beginning in March).17 Formulation may differ for each hemisphere or remain the
same.

Many different influenza vaccine products are produced annually, by a number of

different manufacturers. These include trivalent inactivated vaccines, quadrivalent
inactivated vaccines, trivalent live attenuated vaccines, and quadrivalent live attenuated
vaccines. Inactivated vaccines may be whole-virus vaccines, split virus vaccines
(where the whole virus has been disrupted by detergent), or subunit vaccines (where
the HA and NA components have been further purified from other virus antigens).
Inactivated vaccines may also be produced in high-dose formulations or adjuvanted
formulations. For any vaccine type, each manufacturer’s formulation has its own
safety profile and age group indications (which depend on country-level licensure
and can change over time). VE is also expected to vary across vaccine products; for
example, in elderly adults high-dose vaccines are expected to have greater VE than
standard dose vaccines.18 At present, data are insufficient to assess whether VE varies
across vaccine products within a given class, such as standard-dose non-adjuvanted
inactivated vaccines. Pooled VE estimates across vaccine products suggest that
influenza vaccines reduce the risk of medically attended influenza virus infection by
approximately 50%, although there is considerable heterogeneity by season, setting,
and population subgroup.8,19-21

Influenza viruses vary in transmissibility and virulence from season to season and
cause a wide range of adverse health outcomes. Influenza vaccines vary in degree of
antigenic match to circulating viruses, and influenza VE can differ by age group, virus
type/subtype, vaccine product, health status of vaccinees, health outcomes under
study and the degree of antigenic match between vaccine and circulating viruses. 22
VE can also differ between groups that have residual immunity to circulating viruses
(from past exposure or prior vaccination) and those naive to circulating viruses.
Public health scientists and others who support influenza VE studies need to be
aware of the variability that is possible in influenza VE estimates. This requires
careful consideration of study endpoints prior to initiating VE studies, and proper
generalization of VE estimates in relation to study participant profiles, types/subtypes
of contributing viruses, study location, and years when studies are completed.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 3
1.4 Organization of this guide
The guide is organized in the following sections:

Section 2 describes the role of influenza VE studies in implementing and evaluating

influenza vaccination programmes.

Sections 3–5 cover outcomes of interest, measuring vaccination history, and

measuring potential confounders – topics of importance for any influenza VE study.

Section 6 reviews different designs that could be used for influenza VE studies, and
the strengths and limitations of each.

Section 7 reviews statistical considerations for designing VE studies and for analysing
VE study data.

Section 8 provides a more detailed description of one specific design that is most
likely to be attempted in low-resource settings where influenza vaccines have been
introduced to at least some parts of the population: a test-negative study built on an
existing severe acute respiratory infection (SARI) surveillance system.

Section 9 describes key elements that need to be included in any report of an influenza
VE study, so that the results can be properly interpreted.

4 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 2. The role of VE studies
in influenza vaccination
programmes
Two scientific considerations help determine
whether to introduce a particular vaccine in Vaccine Efficacy: Reduced risk of disease
among vaccinated persons resulting
a specific population. The first consideration from vaccination in ideal circumstances;
is the burden of disease that is potentially estimated from randomized trials.
preventable by vaccination. This disease
burden can be assessed by disease surveillance Vaccine Effectiveness: Reduced risk
systems23 or by re-analysis of randomized of disease among vaccinated persons
attributed to vaccination in real-world
clinical trial (RCT) data from vaccine conditions; estimated from observational
efficacy studies as vaccine probe studies for (non-randomized) studies.
selected clinical outcomes. 24,25 The second
consideration is the expected performance of Vaccine Impact: Reduction in incidence
the vaccine in practice. This can be assessed of disease in a population where some
members are vaccinated; usually estimated
through RCTs of vaccine efficacy and through from ecologic (or modeling) studies.
observational VE studies. Observational VE Vaccine impact results from direct effects
studies in countries with existing vaccine of vaccination in the vaccinated, as well as
programmes can provide useful evidence for indirect effects in the unvaccinated which
public health policy-makers in evaluating the are presumed to be due to herd immunity.
expected VE in real-world conditions.

After a vaccine has been introduced in a population, additional studies help in

evaluating the impact of the immunization programme. Data from VE studies can be
used to inform vaccination policy details, such as determining whether the vaccine
is effective in groups at high risk of severe disease or identifying preferred vaccine
product classes. For example, observational influenza VE studies in the United
States during the 2013/14, 2014/15, and 2015/16 influenza seasons led the ACIP to
recommend against the use of live attenuated influenza vaccines for the 2016/17
influenza season. 26

For the purposes of this guide, “impact” refers to reduction in incidence of disease due
to vaccination. The impact of vaccination programmes on disease burden is typically
evaluated using active surveillance programmes that compare the incidence of disease
prior to and after programme implementation.6 For certain vaccines, these assessments
can include both the incidence of laboratory-confirmed outcomes, such as invasive
S. pneumoniae disease, and of non-specific outcomes, such as hospitalizations for
pneumonia, with the difference in incidence attributed to the vaccination programme.
These so-called “before-after studies” applying interrupted time-series methods
to surveillance data can show the degree to which introduction of the vaccine has
reduced disease incidence. While they have been used for other vaccines, including

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 5
pneumococcal, Haemophilus influenzae type B (Hib), and rotavirus vaccines,6,27,28
they are difficult to conduct for influenza vaccines, necessitating alternative
approaches.

2.1 Challenges to studying the impact of influenza vaccination programmes

In contrast to evaluation of other vaccine programmes, evaluating the impact of
influenza vaccination programmes after vaccine introduction generally relies on
repeated influenza VE studies. However, before-after studies to demonstrate vaccine
impact are very difficult to conduct for influenza, for the following reasons:

i) Influenza VE is modest compared to the effectiveness of other vaccines such

as pneumococcal conjugate vaccine (PCV). RCTs suggest that a primary series
of PCV in infancy reduces the risk of invasive pneumococcal disease due to
vaccine serotypes by 70–95%, with protection lasting for several years. 29,30 In
contrast, influenza VE against laboratory-confirmed influenza virus infection
is rarely higher than 60% and may sometimes be 30% or less, with protection
that may wane from one season to the next or within a season.20,31-34 Even with
high vaccine coverage, the expected decline in severe influenza disease will be
substantially less than the decline in pneumococcal disease after implementation
of a vaccination programme, making the impact of influenza vaccine more
difficult to measure. Observing population-level impact is further complicated
by the fact that influenza vaccine coverage is rarely high even in high-resource
settings.35,36 For example, in the United States during 2015/16, where influenza
vaccination was recommended for all persons ≥ 6 months of age, vaccine coverage
was 46% in age-eligible persons, 59% in children aged ≥ 6 months, and 42% in
adults.37

ii) Seasonal influenza epidemics display considerable heterogeneity, driven

by population immunity and by antigenic drift of influenza viruses. This
heterogeneity can affect the timing, incidence, and severity of epidemics; it also
causes variations in the degree of similarity between antigens in the vaccine viruses
and in the circulating viruses, also known as antigenic match. VE can vary from
year to year depending on factors such as the degree of antigenic match between
the influenza strains in the vaccine and the strains that circulate.31 Because of this
heterogeneity, many years of surveillance both pre- and post-implementation
would be needed to properly evaluate the impact of an influenza vaccination
programme on disease incidence. If too few years are included in either the pre-
implementation or the post-implementation periods, the conclusions could be
biased. For example, if the pre-implementation period consisted of two years,
both of which were relatively mild influenza seasons with A(H1N1) or B viruses
predominating, the average burden of disease pre-implementation would be
underestimated, making it difficult to demonstrate a decline in incidence post-
implementation. Such studies have not been attempted even in high-resource
settings with moderate vaccine coverage.

Beyond the impact of vaccination on burden of illness, policy-makers may wish

to evaluate the economic impact of vaccination programmes. Economic impact is

6 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
estimated based on rates of outcomes averted at the population level and on estimates
of cost of care. WHO is currently developing guidance on estimation of the economic
impact of influenza vaccination programmes.38

2.2 Use of VE studies to estimate impact of influenza vaccination programmes

Because of the heterogeneity in influenza seasons and the modest effectiveness of
influenza vaccines, interrupted time series studies are unlikely to be feasible for
assessing the impact of influenza vaccination programmes. Instead, influenza
vaccination policy-makers typically have to combine data from multiple sources to
estimate vaccine impact. The first step is to use influenza surveillance data to estimate
the disease burden in terms of influenza-attributable hospitalizations, deaths, and /
or (occasionally) outpatient clinic visits.39 For example, a hospital-based surveillance
system could attempt to enrol all patients meeting specified clinical criteria and test
those patients for influenza. Collecting these surveillance data over multiple years
would allow policy-makers to estimate the incidence of hospitalizations caused by
influenza. The second step is to estimate influenza vaccine coverage among groups
targeted for vaccination. The final step is to estimate influenza VE, which is often
estimated on an annual basis due to the year-to-year variation in VE, particularly in
settings with well-defined influenza seasonality.

Estimates of disease burden, vaccine coverage, and VE are then combined to provide
estimates of the cases, hospitalizations, and deaths averted by vaccination. Among
those vaccinated against influenza, the cases averted per 1000 vaccinees can be
estimated as follows:
I
–I
1 – VE p
Cases averted =
p

Where I is the cumulative incidence of influenza events per 1000 population, p is the
vaccine coverage, and VE is the vaccine effectiveness. If vaccine coverage data are
available by month, then more complicated modelling approaches are also possible.40

Influenza VE studies have several important uses. Initial VE studies after the first
introduction of influenza vaccine can verify that the vaccine works as predicted. Then
ongoing VE studies are used as one input in models of outcomes averted by vaccination.
To this end, many countries and regions have established platforms for ongoing
annual estimates of influenza vaccine effectiveness, such as the European I-MOVE
consortium,8 US Flu VE Network, 20 the Canadian Sentinel Practitioner Surveillance
Network (SPSN), 21 the Pan American Health Organization’s REVELAC-I network,7
and the Australian FluCAN vaccine effectiveness surveillance.41,42 The VE estimates
from these platforms are used to estimate the health impact of vaccination.43,44 Given
the lack of a robust correlate of protection, the European Medicines Agency (EMA)
has recently begun requesting observational studies as part of the manufacturer`s
Risk Management Plan.45 The I-MOVE network and the European Centre for
Disease Prevention and Control (ECDC) have useful websites with generic protocols,
research publications, and reviews.46-49 

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 7
 3. Outcomes of interest for
influenza VE studies

A number of outcomes have been used for

influenza VE studies, reflecting the range Outcome: A specific, measurable
of morbidity and mortality that influenza disease or health event. All study subjects
virus infection can cause and which may should be at risk for the outcome, and
the occurrence or non-occurrence of the
be preventable by vaccination. This section outcome should be known for all subjects.
reviews several of these outcomes and
evaluates the benefits and limitations of the Outcome misclassification: Incorrectly
use of each. As with other vaccine preventable determining the occurrence of the outcome
diseases, such as pneumococcal pneumonia, for a study subject. This includes falsely
considering a diseased person to be
selecting appropriate VE outcomes involves disease-free, and falsely considering a
a trade-off between providing information disease-free person to be diseased.
that is of greatest interest to policy-makers
and identifying outcomes that can be assessed Study participants: All persons enrolled in
a VE study. Outcomes, vaccine history, and
with a minimum of misclassification or other
covariates should be measured on all study
type of bias. For the purposes of VE studies, participants.
outcomes can be classified according to two
major considerations: the specificity for
influenza virus infection, and the severity of
the illness.

i)  ith respect to specificity, influenza vaccine is only effective against influenza

W
virus and not against diseases that are not directly or indirectly caused by
influenza virus infection. Laboratory-confirmed influenza illness is a highly
specific influenza outcome. Even during the peak of the influenza season, the
non-specific clinical syndrome of “influenza-like illness” (ILI) includes various
other respiratory pathogens that cause a similar syndrome against which
influenza vaccine does not protect.50 The proportion of ILI not due to influenza
varies by intensity of influenza and other respiratory virus activity, so while VE
against ILI will always be lower than VE measured against laboratory-confirmed
influenza outcomes, the extent of under-estimation will vary.

ii) Influenza outcomes can be classified by severity, ranging from mild illnesses
managed at home without any intervention or attended on an outpatient basis, to
more severe outcomes of hospitalization, ICU admission or death.

8 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
The non-specific clinical presentation of influenza infection and the broad range
of potential influenza complications create challenges for determining appropriate
outcomes for influenza VE studies, as described below.

i) I nfluenza virus infection can cause severe illness and death through a number
of different pathways and clinical syndromes. Infected individuals can have
severe influenza virus infections, generally characterized by cough and difficulty
in breathing, reflecting lower respiratory involvement.51,52 Influenza can also
cause severe illness by exacerbating existing chronic disease such as asthma
or chronic obstructive pulmonary disease (COPD). For these patients, typical
influenza disease signs such as fever may be absent. Furthermore, influenza virus
infection can predispose individuals to secondary bacterial pneumonia. In such
cases, the time from initial influenza virus infection to secondary pneumonia
to hospitalization may be long enough that patients no longer shed detectable
influenza virus when they present for medical care. Influenza virus infection
may also trigger cardiopulmonary complications including acute myocardial
infarction; patients presenting with these conditions are unlikely to be tested
for influenza virus infection.53 Due to the variety of potential complications of
influenza virus infections, it is difficult to select outcomes for VE studies that
capture the range of influenza-associated complications. Selecting only one
outcome for a VE study (such as pneumonia) may underestimate the vaccine
benefits because other important outcomes that influenza vaccine may prevent,
such as exacerbation of COPD and cardiovascular disease, are overlooked.

ii) M
 ost of the clinical syndromes that influenza virus infection may cause can
also be caused by other factors, including other respiratory pathogens. Severe
respiratory disease may be caused by influenza, respiratory syncytial virus, and
adenovirus, among others.54 For any given clinical syndrome, such as severe
acute respiratory infection (SARI) or ILI, the contribution of influenza to
the total burden may range from moderate to low and will differ from year to
year and between different locales. For example, the contribution of influenza
to hospitalized pneumonia cases may range from <1% to 12% of such cases.55
Because the majority of cases of these clinical syndromes are caused by pathogens
other than influenza virus, an influenza vaccination programme will not lead to
major reductions in the incidence of syndromes with multiple causes, such as
pneumonia, or use of primary or secondary health-care services, such as seasonal
increases in emergency department presentation. For example, assuming that
influenza causes 10% of hospitalized pneumonia cases in a given age group and that
influenza VE against influenza is 50%, the expected VE against the syndrome of
pneumonia would be only 5% (50% prevention of 10% of pneumonia cases).56 A
vaccine with a low VE against a serious and relatively common clinical syndrome
might be of great public health value. However, detecting a low VE requires a
very large sample size and a correspondingly large research investment, as well
as reasonable uptake of vaccine in the target population.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 9
The following are some clinical outcomes that could be used as endpoints for influenza
VE studies:

1. Non-specific (i.e. syndromic) outcomes

A. S evere acute respiratory infection
The WHO Global Epidemiologic Surveillance Standards for Influenza define
SARI as an acute respiratory infection with history of fever or documented
fever of ≥38 oC, cough, onset within the last ten days, and requiring
hospitalization.39 SARI offers several advantages as an outcome for influenza
VE studies. Patients with SARI are by definition hospitalized and typically
have severe illness. Depending on the stated programme objectives, this
outcome is likely to be of particular interest to policy-makers, since severe
illnesses are much more important in terms of financial and health costs (e.g.
disability-adjusted life years) than mild illnesses. As many countries have
existing systems of SARI surveillance, it may not be necessary to set up
new surveillance systems to identify SARI cases for influenza VE studies,
reducing the potential cost of VE studies.

SARI also has important limitations as an endpoint for observational

influenza VE studies. Foremost is the fact that influenza makes at most only a
modest contribution to the overall incidence of SARI. A two-year study in the
Philippines, for example, found that only 9% of SARI cases had laboratory-
confirmed influenza virus infection.57 Even if influenza vaccine were 100%
effective against laboratory confirmed influenza, an influenza VE study using
SARI as an endpoint would find 9% effectiveness at most in this setting. An
observational study would require a large sample size (and likely many study
sites) to measure effectiveness with adequate precision. Another limitation is
that, in some settings, influenza vaccine may be used disproportionately by
persons with a different risk for SARI than unvaccinated persons (e.g. those
with chronic heart disease). Differential vaccine utilization can cause very
strong confounding of VE estimates in observational VE studies.58

B. All-cause pneumonia requiring hospitalization

All-cause pneumonia requiring hospitalization has many of the same benefits
and limitations as SARI as an endpoint for influenza VE studies. VE against
severe outcomes is of high interest to policy-makers, and some countries have
existing surveillance systems for hospitalized pneumonia cases, particularly
among children <5 years of age. Pneumonia has the additional advantage of
being a clinical diagnosis with which clinicians are familiar. However, the
contribution of influenza to pneumonia is low, and observable VE against
pneumonia will also be low. Furthermore, many influenza-associated
pneumonia cases occur in adults aged at least 50 years, cases which are often
not captured by existing pneumonia surveillance systems focused on children.
Thus, using hospitalized pneumonia as an outcome will likely require study-
specific surveillance systems to be developed. Pneumonia is also subject to
confounding in the same way as is SARI.

10 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 C. Influenza-like illness
ILI is defined by WHO as an acute respiratory infection with measured fever
of ≥38 oC, cough, and onset within the last seven days.39 This definition was
created primarily for the purpose of conducting virologic surveillance, i.e. for
identifying the occurrence of influenza and collecting specimens which were
likely to test positive for influenza virus. ILI as a diagnosis is typically applied
to patients in outpatient settings. Outpatient ILI has several advantages for
VE studies. It is relatively easy to apply the ILI case definition in setting
up outpatient surveillance. At least in regions with well-defined influenza
epidemics, the proportion of patients (during the epidemic period) who have
influenza is higher among patients with ILI than among patients with SARI
or pneumonia. This greater specificity of the outcome leads to larger VE
estimates relative to SARI or pneumonia, which reduces the relative sample
size needed to detect statistically significant VE.

ILI as a VE endpoint has several disadvantages. The main limitation is that

illness treated on an outpatient basis is less compelling to policy-makers in
developing countries, for whom influenza vaccine programmes are likely
to be justified based on the expected impact on serious outcomes. While
demonstrating VE against ILI would indicate that the vaccine can reduce
the risk of influenza-associated outpatient illness, VE against ILI cannot
be extrapolated to VE against more serious outcomes without making
assumptions that are difficult to validate.

D. All-cause mortality
All-cause mortality was formerly a common endpoint for observational
influenza VE studies, particularly among older adults.59,60 More recent studies
from the United States have shown that among older adults, those at higher
risk for death are less likely to receive influenza vaccine than those at lower
risk for death, and that these differences in risk lead to strong confounding in
VE estimates, often called the “healthy vaccinee bias”.61,62 It is very difficult
to accurately estimate VE in view of this confounding, even when using
sophisticated statistical methods and extensive medical record data.63 All-
cause mortality is not recommended as an endpoint for VE studies.

E. Adverse birth outcomes

In recent years, increasing attention has been paid to the possible benefits
of maternal influenza vaccination on birth outcomes such as small for
gestational age and low birth weight. Influenza virus infection in pregnant
women may sometimes result in preterm birth, although the evidence for this
is not consistent. If maternal influenza virus infection can cause poor birth
outcomes, then influenza vaccination may prevent some of these adverse
outcomes. Maternal influenza vaccination can also protect neonates and
young infants from influenza virus infection.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 11
 A reduction in adverse birth outcomes or infant influenza virus infections
would be an important public health benefit of influenza immunization
programmes. However, WHO recommends caution with the use of adverse
birth outcomes as endpoints of observational influenza VE studies, for several
reasons:

i)  major methodologic challenge in the study of adverse birth outcomes

A
is the difficulty of accounting for the interrelated effects of calendar time
and gestational age. Both the receipt of influenza vaccination and the
periods of risk for influenza virus infection depend on calendar time (on
the availability of vaccine and on the circulation of influenza viruses,
respectively), and the timing of vaccination relative to gestational
age depends jointly on the availability of vaccine and on the date of
conception. Failure to properly account for these joint effects can bias
VE estimates.

ii)  onfounding due to differences in risk of adverse outcomes between

C
vaccinated and unvaccinated women is a strong possibility. Women at
increased risk of preterm birth or other outcomes may be preferentially
offered or seek vaccination. Unless study staff have access to relevant
medical records or other data for identifying women at higher risk of
adverse outcomes, this confounding will be very difficult to control.

iii) A
 review of observational study design methodologies for studies
assessing influenza vaccine association with adverse birth outcomes
provides further rationale for recommending caution with the conduct
and interpretation of these studies.64,65

For any non-specific outcomes, assessment of influenza VE should be

restricted to time periods when influenza is circulating in the source
population. Enrolling subjects only during periods when influenza circulates
will maximize the proportion of non-specific outcomes that are caused by
influenza, which will in turn increase the ability of the study to detect any
vaccine effects. Caution must be used in reporting these results, however, so
that it is clear that the estimated VE is against the outcome during periods
of influenza circulation only (i.e. when the outcome is more likely to be true
influenza), not against the outcome year-round (i.e. when other causes of the
non-specific outcome are more contributory).

2. Laboratory-confirmed
 outcomes
RT-PCR is the standard test for laboratory confirmation of influenza virus
infection during acute illness. RT-PCR and other commercially available
molecular diagnostic tests are highly sensitive and highly specific for detecting
influenza viruses. Although other assays are available, including viral culture,
rapid antigen tests, and immunofluorescence, these are less sensitive than RT-

12 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 PCR. Rapid antigen tests are also considerably less specific than RT-PCR and can
cause substantial misclassification, as imperfect specificity is more likely to bias
VE estimates than imperfect sensitivity. 22,66 Rapid antigen tests have the further
disadvantage of being unable to distinguish between influenza A subtypes or (for
some tests) between influenza types A and B. Beyond laboratory confirmation
of the presence of influenza virus infection, detection of the specific influenza
type/subtype by RT-PCR allows researchers to stratify influenza VE estimates
by virus type/subtype, as discussed in more detail in Section 8.

In general, laboratory-confirmed outcomes have several advantages over non-

specific outcomes for influenza VE studies. Laboratory-confirmed outcomes are
much more specific for influenza virus infection than are outcomes based on
clinical signs and symptoms. Thus, VE estimates will be higher (and it will be
easier to detect significant VE) when using laboratory-confirmed outcomes.

A major constraint with the use of laboratory-confirmed outcomes in influenza

VE studies is the need for appropriate laboratory capacity, including specimen
collection, handling, and storage as well as molecular assay technology and
reagents. All of these activities require training of clinical staff to ensure
standardized specimen collection and handling. Training of laboratory staff
is also required to ensure proper processing and testing of specimens. This is
particularly critical with the use of RT-PCR technology, which can give false
positive results if specimens become contaminated during processing. Improper
specimen handling or collection of inappropriate specimens can lead to false
negative results. Furthermore, due to the need to enrol patients who will likely
have detectable influenza virus, laboratory-confirmed outcomes are typically
restricted to patients who present for illness within seven days of illness onset.
This excludes patients who may have developed severe disease, such as secondary
bacterial pneumonia, more than seven days after illness onset.

The decision to assess potential study subjects for laboratory-confirmed

influenza virus infection should not be based on clinicians’ decisions but on pre-
specified protocol guidelines. Clinical decisions to test specimens for influenza
can introduce bias into VE studies in a number of ways. Most obviously, when
clinicians have access to vaccination records they may be less likely to order
influenza testing in vaccinated patients, reasoning that vaccinated patients are less
likely to have influenza than unvaccinated patients. This would falsely lower the
number of vaccinated cases among the study participants and bias VE estimates
upwards. Even in the absence of vaccination data, clinical testing decisions may
introduce bias in more subtle ways. Among patients with acute respiratory illness
(ARI), US physicians preferentially prescribe influenza antivirals to patients
who are later found to test positive for influenza.67 This suggests that physicians
have some ability to distinguish influenza ARI from ARI due to other causes.
Physicians may preferentially order testing of patients they suspect of having
influenza (to validate their suspicion), or testing of patients whose influenza

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 13
 status they are unsure of (to address their uncertainty). Either case may cause
systematic misclassification of the potential pool of study subjects. To avoid these
biases, study protocols should specify the symptoms, duration of illness, and
other eligibility criteria for attempting to enrol and test patients for influenza.
These criteria should then be applied to all (or to a random sample of) eligible
patients regardless of clinical testing preferences.

Laboratory-confirmed outcomes, including both outpatient outcomes20 and

hospitalized outcomes,68 are becoming standard for VE studies in high-resource
countries. WHO recommends the use of laboratory-confirmed outcomes whenever
possible because of the benefits associated with use of outcomes specific for influenza.

Recommended background research on outcomes to inform VE studies:

1) If non-specific outcomes such as SARI hospitalizations, pneumonia, or ILI are to be used, background research
should estimate the incidence of these outcomes and the expected proportion that are caused by influenza.
These data will be needed for sample size calculations to determine the feasibility of a VE study. They will
also allow researchers to estimate the range of VE estimates that would be expected from a VE study. For
example, if 20% of pneumonia hospitalizations were attributed to influenza in some target populations, VE
against pneumonia is expected to be 20% or less, and should be around 10% if VE against influenza is 50%.

2) If laboratory-confirmed outcomes are to be used, researchers need an understanding of the testing process.
If there is to be active testing of specimens collected from patients meeting a certain case definition, the study
team needs to know how to systematically identify those patients. If the study will rely on clinical testing, the
study team needs to evaluate the clinical decision process to order influenza tests, and particularly what patient
characteristics are associated with clinical testing.

3)  or cohort studies, the researchers need to determine the care-seeking patterns of the proposed study cohort.
F
What facility/facilities will be used for outpatient care? Where will cohort members be hospitalized for severe
illness? What proportion of outcomes may be missed due to seeking care outside of study facilities or self-
treatment at home?

4)  or traditional case-control studies, researchers need to understand the catchment areas of the study enrolment
F
facilities. What is the source population for the study hospitals or clinics? Where else might ill persons in this
population seek medical care? Who are the influenza vaccine providers for this population? This information
will be needed to identify appropriate control groups.

14 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 4. Assessing influenza
vaccination history

As part of an observational influenza VE

study, each subject’s influenza vaccination Current influenza vaccination: Receipt
of one or more doses of the current year’s
history must be ascertained. The ultimate
influenza vaccine.
goal is to assess whether the subject has
received the current season’s influenza Prior influenza vaccination: Receipt of
vaccine prior to any outcomes of interest. one or more doses of influenza vaccine in
This means that both the fact of vaccination previous years.
and the approximate timing of vaccination
Vaccination misclassification: Falsely
need to be determined. Fact of vaccination classifying a vaccinated subject as
must distinguish receipt of the currently unvaccinated, or falsely classifying an
recommended influenza vaccine from unvaccinated subject as vaccinated, or
influenza vaccination in previous influenza falsely classifying person-time prior to
vaccination as being post-vaccination.
seasons. For most influenza vaccines, it is
recommended that children aged <9 years
who are vaccine-naïve should receive two doses separated over time when they are
vaccinated for the first time. For these children, it is desirable to document previous
influenza vaccinations to determine whether they are fully or partially vaccinated. In
settings where multiple vaccine products may be in use, the type of vaccine product
(e.g. inactivated, live, adjuvanted, high-dose) used should be documented if possible,
ideally specifying actual product name.

Timing of vaccination is important to make sure subjects are counted as vaccinated

only after they have received vaccine and there has been sufficient time since
vaccination for an immune response to develop (generally assumed to be 14 days
post-vaccination). This is particularly important if vaccination is being carried out
during (rather than prior to) the period of peak influenza virus circulation.

Vaccination history can be assessed in several ways:

i) One approach is to rely on administrative records created as part of vaccination
programmes, such as patients’ vaccination cards. Many vaccination programmes
in low-resource settings focus on vaccines given to children <5 years of age, such
as measles, mumps, and rubella vaccine and diphtheria, tetanus, and pertussis
vaccine. Most countries have some established infrastructure for administering
these vaccines and for keeping vaccination records, such as vaccination cards
kept by parents, although influenza vaccine receipt is often not included in
vaccination cards in many low-income countries. Many of the persons at high
risk for influenza-associated complications are adults. Vaccination cards are not

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 15
 usually given to adults, and without an established infrastructure, influenza
vaccination may not be routinely recorded in clinic records, requiring reliance
on self-reporting.

ii)  nother approach is to assess influenza vaccination status through national

A
or provincial vaccination registries, either electronic immunization registries or
paper EPI registries.69 Where available, these registries can greatly facilitate the
determination of the vaccination status of study subjects. However, registries
are not perfect. In some settings it may be difficult to match a study subject to
a record in the registry and data entry errors can lead to incorrect information
in the registries. Even in high-income countries, data in vaccine registries are
generally only complete for registry items that are required for entry of a record
into the database (such as patient ID and vaccine name); items that are optional
generally have substantial missing data, and it is often these fields that are of
interest to researchers.70 Registry recorded vaccine receipt has high specificity
but variable sensitivity depending on quality controls.

iii) P
 erhaps the most straightforward approach to assessing vaccination history is
to ask subjects whether and when they (or their children, for parents of enrolled
children) received the current year’s influenza vaccine. In some settings, adult
subjects have been able to accurately recall their influenza vaccination status,
relative to vaccination registry records. However, subjects’ recall of influenza
vaccine history can be inaccurate, particularly for parental report of a child’s
vaccine history.71-75 Eliciting self-reported vaccine history from subjects involves
an interaction between subjects and research staff. A measurement error can
arise due to researchers not being blinded to subjects’ case status (in a case-
control study) or due to subjects’ desire to please the researcher. An additional
limitation of self-reported vaccination is that subjects’ recall will not capture
information on the vaccine product received, if vaccines from multiple suppliers
or of multiple formulations are available in a geographic region (although this
would only be a concern where finer exploration of vaccine product-specific VE
estimates are desired). Due to these limitations, WHO recommends against the
use of self-reported vaccination history as the sole source of vaccination data.

Any method of assessing influenza vaccination history in an observational study

has potential pitfalls and limitations. Ideally, influenza vaccination programmes,
particularly new ones, should ensure a systematic way to record vaccine history so that
the information is easily accessible. If record-keeping related to influenza vaccinations
can be designed with subsequent observational studies in mind, the likelihood of
successfully executing those observational studies will be much greater. In any event,
it is essential to conduct pilot testing methods for collecting vaccine history before
beginning any influenza VE study. This pilot testing should assess the accuracy
and completeness of any proposed method for measuring vaccination history, and
study protocols should be adapted accordingly. In some cases, incorporating multiple
approaches for assessing vaccination status may be the best way to ensure that vaccine

16 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
information is complete and accurate. The manner of exposure ascertainment and
the results of pilot testing should be reported to allow readers to put the VE results
into context and assess the strength of potential biases. In particular, these results
should include the sensitivity, specificity, and positive/negative predictive values of
the proposed approaches to defining subjects as vaccinated or unvaccinated.

Recommended background research on vaccination history to inform VE studies:

1) If a proposed study is to rely on a vaccination registry, pilot testing should assess the ability of study staff
to link study subjects to the registry, and should attempt to evaluate the accuracy and completeness of the
registry data.

2) If a proposed study is to rely on vaccination cards, pilot testing should assess the ability to review the cards
and the legibility and validity of the data on the cards.

3) If a proposed study is to rely on self-reported vaccination history, pilot testing should validate self-reported
vaccinations relative to some external standard. All studies should assess vaccine coverage among
participants (notably controls) in relation to expected values in the source population as a check on possible
bias in vaccination status.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 17
 5. Measuring covariates

In addition to measuring outcomes and

Bias: Systematic deviation of study findings
ascertaining vaccination history on all VE from true associations in the source
study participants, investigators will also population.
need to collect data on other covariates.
One purpose of measuring covariates is Selection bias: Bias resulting from
to stratify VE estimates based on sub- differences in enrolling or in collecting data
on study subjects based on vaccination
populations of interest. Depending on history or outcome status.
the objectives of the evaluation, existing
influenza VE programmes often produce Confounding: Bias arising when exposure
stratified VE estimates, e.g. by age and outcome share a common cause that is
group. 20,68 Other groups of interest might not taken into account in analyses.
be pregnant women or persons with certain Covariates: Factors other than outcome
comorbidities. and vaccination history that are measured
on all study subjects. Examples include age,
sex, and comorbid illnesses. Covariates
The other main purpose for collecting
are used to produce stratified VE estimates
covariate data is to measure and control and to control for confounding. Detailed
for potential confounders, either in study reporting of participant profiles based on key
design (e.g. by matching) or in data analysis. covariates also supports understanding of
In observational VE studies, confounding the applicability of results across seasons/
settings and by comparison to expected
due to differences in disease risk or in care-
values in the source population supports
seeking behaviour between vaccinated assessment of possible selection bias.
and unvaccinated subjects can greatly bias
VE estimates. In an extreme case, studies Potential confounder: A covariate
using all-cause mortality as an outcome associated with both vaccination and
outcome and not caused by either.
and without measuring true confounding
Confounding in VE estimates may be
factors may produce VE estimates that are reduced by including potential confounders
ten-fold greater than what is expected or in statistical models of VE.
even plausible.58,59,76-78

The important confounding factors to measure will depend on the outcome being
studied, the study setting, and the target groups of the vaccination programme being
studied. Specific variables to measure will represent a compromise between what is
theoretically important to measure and what is possible to measure given constraints
of time and resources. In influenza VE studies, strong confounders have often
been factors that are not readily measured using pre-existing data sources such as
administrative health-care databases.58 This is particularly true of so-called “healthy
vaccinee” effects in older adults, in which frailty is associated with higher risk of
outcomes and lower probability of vaccination, while being difficult to measure

18 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
using health-care databases.61,76 Because of this, measuring important confounders
will likely be time-intensive, requiring either interview of patients or review of
medical records. WHO recommends pilot studies to identify factors that are strongly
associated with vaccination and with disease risk prior to beginning any influenza
VE study in a new setting. This formative research will be essential to the successful
conduct of influenza VE studies.

To guide the selection of covariates to be measured, a number of potential confounders

that have been considered in high-resource settings are described below. Factors that
may be important predictors of both vaccine receipt and of care-seeking among
persons with ARI are also included. The specific covariates to include will depend on
the locale and context of a particular study. For illustrative purposes, the following
list separates the key covariates, which have almost universally been found to be
important for VE studies, and other covariates, which may be useful for specific
settings.

5.1 Key covariates

Age is almost certainly a confounding factor, as both vaccine coverage and risk of
influenza virus infection vary by age. Age is also an important stratification factor for
VE estimates, as VE may differ in different age groups. Stratification of VE estimates
allows researchers to assess the presence of confounding by age or of true difference
in VE by age (i.e. effect modification).

Sex has not often been a confounder in studies conducted in high-resource settings,8
but may be more strongly related to health care utilization and vaccination in other
settings.

Race/ethnicity is correlated with health care utilization in many parts of the world.

Date of symptom onset is an important covariate for characterizing the influenza

epidemic in the population. It is needed in cohort studies to calculate person-time
at risk, and in case-control studies to sample controls (if using incidence-density
sampling).

Calendar time is a key covariate in test-negative studies (described in Section 5),

because non-cases which are enrolled outside of an influenza season must be excluded
from analyses to avoid bias.79,80 Calendar time is also correlated with vaccine uptake
and with incidence of influenza, creating potential confounding by calendar time,
although this confounding may not be meaningful in some settings.81 Date of
symptom onset may be used to adjust for potential confounding by calendar time. It is
important to note that in regions with temperate climates and well-defined influenza
seasons, it is straightforward to determine which subjects to include and which to
exclude. In regions with tropical climates where influenza may have multiple peaks,
it can be more difficult to know when influenza virus is not circulating widely. At
a minimum, calendar time should be considered as a confounder, and sensitivity
analyses should compare VE estimates including all subjects vs only those enrolled
during influenza peaks.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 19
Time from symptom onset to specimen collection may be associated with the
sensitivity or specificity of influenza testing. Stratifying VE estimates by time from
onset to collection can help assess whether this may lead to biased estimates.

Use of antivirals is a confounder as patients who have used antiviral medicines either
for treatment or for prophylaxis are more likely to have false negative test results.
These patients should be excluded from study enrolment.

5.2 Other covariates

Receipt of other vaccines, such as pneumococcal vaccines, may be a marker for
care-seeking behaviour and/or propensity to seek influenza vaccination. For studies
using pneumonia as an outcome, receipt of pneumococcal or Hib vaccines should
be measured as potential confounders, as the risk of pneumonia is lower in those
who have received these vaccines, and their use may be correlated with influenza
vaccination.

Prior history of influenza vaccination, although potentially difficult to measure

accurately in many settings, is useful to measure, as receipt of prior years’ influenza
vaccine may impact the effectiveness of the current season’s vaccine.

Presence and severity of cardiac or pulmonary comorbidities are likely important

confounders. Persons with chronic cardiac or pulmonary disease are at increased risk
of influenza-associated complications if they are infected, and therefore more likely to
become cases in a hospital-based study. In high-resource settings underlying disease is
also correlated with receipt of influenza vaccine, although in a non-linear fashion.63,77
Pilot studies on the association of underlying disease and receipt of vaccine will be
essential for identifying particular aspects of underlying disease to be measured.

Functional and cognitive limitations have also been shown to be important

confounders in VE studies among elderly adults in high-resource settings and
particularly in relation to serious outcomes (i.e. hospitalization).58,77 Their role in
other settings and age groups is less clear, and pilot studies would be useful.

Immunocompromising conditions have generally been uncommon among subjects

included in VE studies in high-resource settings and so have not been important
confounders. However, in settings where the prevalence of HIV/AIDS is high, HIV/
AIDS may be an important confounder to measure.

Socioeconomic status is likely to be highly correlated with vaccination and with

health-care-seeking behaviour, and should probably be measured. This will be an
important area for pilot studies. Some proxy measures, such as household crowding
or number of children in the household, may prove useful, as crowding may capture
both some degree of socioeconomic status and risk of influenza virus infection.

Distance to study hospital/clinic may be correlated both with access to vaccination

and access to medical care.

20 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
Access to medical care will be population-dependent. In some settings, availability
and use of health insurance may affect patients’ ability to seek care at certain facilities.

Other covariates which may be useful include:

Measures of outcome severity, such as duration, subsequent hospitalization

(particularly for outpatient outcomes), or death, may be useful for assessing whether
influenza vaccine reduces severity of outcomes in the vaccinated (although this
is complicated to estimate).82 However, because these occur after the onset of the
outcome, they are not potential confounders and should not be included as covariates
in VE models.

Recommended background research on covariates to inform VE studies:

1) Pilot studies should determine the key predictors of influenza vaccination in the proposed study population. Is
vaccination influenced by geography, socioeconomics, comorbidity, pregnancy, and/or age?

2) Pilot studies should determine the key predictors of care-seeking among persons with influenza or ILI. What
factors influence decisions to seek care, and the facilities at which care is sought? Does care-seeking vary by
geography, socioeconomics, comorbidity, pregnancy, and/or age?

3) Publishing the results of these pilot studies will increase the confidence of readers that any VE study in a new
population has carefully assessed potential confounding factors before embarking on VE estimation.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 21
 6. Study designs

Observational VE studies use a variety of designs, including cohort designs, case-

control designs, and pseudo-ecologic designs such as the screening method. This
section reviews different designs for estimating influenza VE, and their strengths and
limitations.

6.1 Cohort studies

A cohort study is perhaps the most familiar and intuitive observational study design.
In a cohort VE study, subjects are first separated into vaccinated and unvaccinated
groups. In each of these groups (or cohorts), the researchers subsequently attempt
to identify all persons who have the outcome of interest. The incidence rate ratios
or risk ratios are then estimated by comparing the incidence of outcomes among the
vaccinated to the incidence of outcomes in the unvaccinated groups. The risk/rate
ratios are then used to estimate VE:

VE = 100% ( 1 – II ) v

where Iv is the incidence (either incidence rate or cumulative incidence) in the

vaccinated group and Iu is the incidence in the unvaccinated group.

Cohort studies have several advantages relative to other observational study designs.
Due to the intuitive nature of the design, results of cohort studies can be relatively
easily communicated to policy-makers and other stakeholders. In addition, because
cohort studies can directly estimate incidence rates, these studies can be used to
estimate the burden of influenza in the vaccinated and unvaccinated populations and
to estimate the number of cases averted by vaccination. Particularly in high-income
countries, it is often possible to define cohorts and identify influenza vaccination
history and certain covariates through electronic databases from national or corporate
health plan membership.

22 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
Cohort studies also have some important limitations with regard to the estimation of
influenza VE, as follows:

i)  esearchers need to be able to enumerate cohorts of vaccinated and unvaccinated

R
subjects. For prospective cohorts (where cohort members are identified prior to
vaccination), the research team must be able to identify which cohort members
may subsequently receive vaccine. For retrospective cohorts (where cohort
members are identified after vaccination), the research team needs cohorts of
vaccinated and unvaccinated persons in which vaccine status has been determined
independently of any influenza outcomes.

ii)  esearchers must be able to identify the study outcomes in both the vaccinated
R
and unvaccinated cohorts. This can be particularly challenging if study cohorts
are drawn from large urban populations, where cohort members could seek care
from a number of different providers or institutions. Careful thought must be
given to the selection of cohort sites to ensure that there is a high likelihood that
cohort members with the study outcome (such as hospitalization for laboratory-
confirmed influenza virus infection) will be identified. It is particularly
problematic if the probability of an outcome being correctly identified differs
between the vaccinated and unvaccinated cohorts. As an extreme example, if the
vaccinated cohort members are all selected from vaccinated persons in a semi-
rural region with only one or two hospitals, while unvaccinated persons are
taken from an urban region with many hospitals, hospitalizations will be more
easily identified for the vaccinated subjects; failure to identify hospitalizations
in unvaccinated subjects will bias VE estimates downward.

iii) A
 s with all observational studies, cohort studies are susceptible to confounding
due to differences in vaccinated persons compared to unvaccinated persons.
Confounding due to differences in health-care seeking behaviour, or to
differences in risk of severe disease, can be substantial and can completely
overwhelm a true vaccine effect.78

iv) A
 nother limitation for cohort studies is the expense involved. Most of the
outcomes of interest (such as hospitalization for laboratory-confirmed influenza)
are relatively rare. A cohort study will have to involve a large number of subjects,
generally several thousand in each cohort, in order to detect a statistically
significant effect of vaccination. When coupled with the expense of correctly
identifying vaccination history on all study subjects and of conducting thorough
follow-up for events, the cost of a well-designed cohort study can approach the
cost of a randomized trial. Given that a randomized trial is much less susceptible
to confounding, there is little reason to conduct cohort studies of influenza VE,
unless they can be built onto existing population-based studies, or use a special
population for which vaccination status and study outcomes can be identified
accurately. For example, a study of monovalent influenza A(H1N1)pdm VE
among health-care workers in Kenya83 found that even with easily identifiable
outcomes and exposures, cohort studies are still prone to biases due to differences
in health-care seeking between vaccinated and unvaccinated subjects.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 23
6.2 Case-control studies
Case-control studies are also commonly used to estimate influenza VE. In a case-
control design, researchers first identify individuals who experienced the outcome
of interest (cases), and a comparison group of individuals who did not experience the
outcome of interest (controls). Vaccination history is then determined for all cases
and controls, and the odds of vaccination in each group is calculated. VE is estimated
from the ratio of these odds:

VE = 100% ( 1 – OO cases
controls
)
where Ocases is the odds of vaccination among the cases, and Ocontrols is the odds of
vaccination among the controls.

The primary advantage of case-control studies is that they are more efficient than
cohort studies in terms of the number of study subjects required. Health-care
encounters for influenza (and particularly for severe complications of influenza) are
rare enough that a cohort study may require dozens or even hundreds of subjects
to identify each case. In contrast, a case-control study may only need three or four
controls per case to achieve comparably precise VE estimates.

Case-control studies also present important challenges:

i)  isclassification of vaccine history is an important source of bias in case-

M
control studies. A prospective cohort study identifies population members
based on available vaccination history data, reducing the risk of misclassifying
vaccine history. Because subjects are not identified and enrolled on the basis
of vaccination status, case-control studies require investigators to determine
vaccination history retrospectively for all cases and controls after they have been
enrolled in the study. In the absence of registries or electronic medical records,
it can be difficult to assess vaccination history accurately.

ii) S election bias is also a potential problem. In high-resource settings, where patients’
vaccination history may be readily available to health-care providers and testing
criteria are not determined by a study protocol, physicians may be more likely to
order testing for influenza if the patients are unvaccinated. Studies of influenza
VE should not allow influenza testing to be based on vaccination status. Ideally,
testing of clinical specimens for influenza should be based on standardized case
definitions and not on clinical decision-making which can introduce bias. If
clinician-based testing is used, absence of bias should be demonstrated.

iii) T
 he choice of an appropriate control group is the greatest challenge in case-control
studies. The controls should be chosen so that the distribution of vaccination is
the same among the controls and in the population that gave rise to the cases.
It can be very difficult to identify a proper control group. For instance, for a
case-control study in which cases are patients hospitalized for laboratory-
confirmed influenza, what is an appropriate control group? One common
approach is to randomly select asymptomatic (and presumably disease-free)

24 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 individuals from the community around the hospital. However, the hospital
may have a wide catchment area, such that hospitalized patients may come from
a variety of communities, some at a considerable distance from the hospital.
Selecting controls from the community around the hospital will lead to biased
VE estimates if vaccine distribution or outcome risks differ across communities.
Matching controls to cases based on community of residence can reduce this
bias, but can be logistically challenging. Alternatively, hospital-based controls
could be used by selecting inpatients at the same hospital who are hospitalized
for a disease other than influenza. Such persons are likely to come from the same
distribution of communities as the persons hospitalized for influenza, making
them a more representative control group. Care must be taken in selecting these
controls, however, because receipt of influenza vaccine may be correlated with
receipt of other vaccines. The control group should not be hospitalized for a
vaccine-preventable illness; otherwise, the prevalence of influenza vaccination
in the controls will not represent the source population for the cases.

6.3 The test-negative design: a special instance of case-control evaluation

A third option for selecting controls, and one that is increasingly used for annual
influenza VE estimation, 22 is to use patients who meet the specimen collection criteria
from the study protocol, are tested for influenza, and are found to have negative test
results. In this approach, the target population for enrolment consists of all persons
who seek care for a defined set of symptoms, typically ARI; cases are those with
positive tests for influenza, and non-cases are those with negative test results. VE is
then calculated as:

VE = 100% ( 1 – OO ) pos
neg

where Opos is the odds of vaccination among those testing positive for influenza, and
Oneg is the odds of vaccination among those testing negative.

The test-negative design is predicated on the core assumption that influenza vaccine
only protects against influenza, and has no effect on other non-influenza causes of
ILI,79,84 a core premise that has been validated through randomized controlled trial
data sets.85 It is powerful for several reasons. Firstly, all cases and controls have
sought care at the same facilities. Hence cases and non-cases will generally have come
from the same communities, reducing bias due to community-level variations in
vaccine coverage. Secondly, cases and non-cases have all sought care for similar sets
of symptoms. This reduces confounding due to differences in health-care seeking
behaviour between cases and non-cases, which is a major challenge to influenza VE
studies. Vaccine status is typically collected and recorded at the time of specimen
collection, prior to knowing the influenza test result, reducing the likelihood of
differential exposure misclassification. Even with sensitivity for influenza detection
as low as 70%, in the context of near-perfect specificity such as provided by influenza
laboratory-confirmed by PCR, outcome misclassification has been shown to have
negligible impact on VE estimates derived by the test-negative design.66

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 25
While the test-negative design has a number of important strengths, caution is needed
in the use of test-negative studies for influenza VE against severe outcomes, such as
laboratory-confirmed influenza SARI. Numerous published studies have shown that
test-negative designs effectively reduce confounding in studies of outpatient illness
in high-resource settings. Similar data for test-negative designs for hospitalized
outcomes are much sparser. As an example of one potential source of bias in test-
negative studies, it is well known that underlying cardiac and pulmonary disease is
strongly correlated with the likelihood of receiving influenza vaccination in high-
resource settings. These underlying diseases also increase the risk of hospitalization
for respiratory symptoms. If the symptoms used to identify patients to be included in
a hospital-based study are too broad, the pool of non-cases could be biased towards
persons with acute exacerbations of chronic conditions, who may have a biased
distribution of influenza vaccination relative to the cases.

6.4 The screening method

The screening method is a pseudo-ecologic design, which uses individual-level
data on vaccination history and other covariates from cases, and ecologic data on
vaccination coverage in the population from which the cases came.86 The advantage of
the screening method is that it does not require detailed data collection on non-cases,
which saves cost relative to a case-control study. However, studies using the screening
method are fully dependent on accurate and valid data on vaccine coverage in the
source population. It is also important to note that the source population for cases
may differ from the general population in which the study is conducted, because of
differences in access to care and health-care seeking behaviour or other reasons. This
requires that the proper source population for the cases can be correctly identified
and that vaccine coverage is properly estimated in this population. In many low-
resource settings, vaccine coverage is calculated based on vaccine doses distributed
and on estimated population size.87

Because estimates of population size can be inaccurate due to seasonal migration

and other factors, vaccine coverage estimates are often inaccurate when applied
to a subset of the general population. Furthermore, it can be difficult to adjust for
potential confounders using this design, as doing so requires separate estimates of
vaccine coverage in population sub-groups defined by levels of the confounding
factors. Studies using the screening method should only be undertaken in settings
where vaccine coverage can be measured with high accuracy, which is a challenge
in many settings. WHO therefore recommends against the use of screening method
designs for estimating influenza VE.

26 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 7. Statistical considerations

7.1 Sample size

As with any epidemiologic study, influenza VE studies must be large enough to rule
out chance as an explanation for the study results. In general, influenza VE studies
are not simply intended to determine that VE is greater than zero, but to estimate
the actual VE with some degree of precision. The sample size needed to estimate VE
will depend on a number of factors, including the proportion of the population that
is vaccinated, the incidence of the study outcome and its specificity for influenza, the
expected VE, and the desired precision of the VE estimates. The effects that some of
these factors can have on study sample size are illustrated in Table 1 and Table 2.

Table 1: Number of cases needed in a case-control study to detect the

specified VE against laboratory-confirmed outcomes, assuming 30% of the
population is vaccinated, 3 controls per case, a type I error rate of 0.05

Precision
(± percentage points) VE = 70% VE = 50% VE = 30%

20 109 215 351

15 192 370 612
10 412 823 1367

Table 2: Number of cases needed in a case-control study to detect

the specified VE against laboratory-confirmed outcomes within ±10
percentage points at the specified vaccination coverage, assuming 3
controls per case, a type I error rate of 0.05

VE 5% vaccinated 10% vaccinated 25% vaccinated 50% vaccinated

10% 9424 4945 2315 1670

30% 6881 3532 1574 1033
50% 4590 2328 967 565

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 27
Table 1 illustrates that increasing the precision of VE estimates requires increasing
the sample size. Estimating a VE of 50% within ±20 percentage points requires 215
laboratory confirmed cases, while estimating 50% VE within ±10 percentage points
requires 827 laboratory confirmed cases. Table 1 also illustrates that, in general, the
closer VE is to zero, the greater the sample size required to demonstrate statistically
significant protection. Estimating VE of 30% within ±15 percentage points requires
612 cases, compared to 370 cases if VE were 50%.

Table 2 illustrates that low vaccination coverage in the population of interest increases
the sample size needed to detect VE, or to estimate VE with a given precision. To
detect a VE of 30% (±10) when 50% of the population is vaccinated would require
1033 cases in a case-control study. If only 5% of the population is vaccinated, detecting
the same VE with the same precision would require 6881 cases.

Also of note is that sample size requirements specified in Table 1 represent what is
needed to estimate average overall VE across (for example) all age groups and for all
virus types and subtypes. Estimating age group, vaccine product, or type/subtype
specific VE will require correspondingly overall larger sample sizes. For example,
estimating VE of 50% within ±10 percentage points in each of four age groups would
require 565 cases in each age group, or 2260 cases overall. Furthermore, adjusting for
confounders (via stratification or regression models) will also increase the sample size
required to estimate VE with a specified precision.

7.2 Characterizing the study participants

Several descriptive analyses are useful for characterizing the study participants.
Graphs of case counts over time are informative for understanding the seasonality
of influenza in the target population.88 These plots can also help in assessing whether
calendar time is a potential confounder, particularly if coupled with plots of vaccine
uptake over time. Flowcharts of the enrolment process can be used to identify possible
sources of bias such as high rates of refusal to participate in the study.

Bivariate descriptive statistics are used to assess the distribution of covariates

among the study participants. These are particularly useful for comparing covariate
distributions between vaccinated vs unvaccinated subjects and between influenza
cases vs non-cases. These bivariate comparisons aid in determining variables that
may be included as potential confounders in adjusted VE estimates, by identifying
variables that are associated with both vaccination and influenza disease. The bivariate
distributions are also useful for comparing the study participants with participants in
other influenza VE studies, as differences in age or comorbidities may contribute to
different VE across populations.

7.3 Analytic approaches to estimate crude rate/odds ratios

In cohort studies, unadjusted (or crude) rate ratios can be obtained from the ratio of
incidence rates of the study outcome between vaccinated and unvaccinated subjects.
Unadjusted incidence rate ratios can be calculated using 2x2 tables or Poisson regression
models. In case-control and test-negative studies, unadjusted odds ratio estimates can
be obtained from the ratio of the odds of exposure among cases vs controls/non-
cases. Adjusted odds ratios can also be calculated using logistic regression models.

28 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
7.4 Assessing potential confounders
In planning the analyses, certain variables will be included in the final VE models
a priori, without examining their actual role as confounders. These variables may
be chosen based on subject-matter knowledge that suggests they will be strong
confounders, or for comparability with estimates from other populations. 22,89 Age,
for example, is associated both with vaccination and with risk of influenza, and so
should be included a priori.

Beyond the variables selected a priori, researchers should determine which measured
covariates are acting as confounders in the data. For this, the most appropriate
method for the final selection of potential confounders to include in statistical models
is the “change-in-estimate” approach.90,91 In the change-in-estimate approach, the
unadjusted VE is estimated using an appropriate statistical model. VE is then estimated
adjusting for a single covariate. If the adjusted VE differs from the unadjusted VE by
more than a pre-determined percent, the covariate is considered to be a confounder
and will be included in final models. A common threshold is to include covariates
whose adjustment changes the crude odds ratio by 10% or more, but the threshold
is at the researchers’ discretion. For example, if the crude odds ratio estimate is 0.56,
after adjusting for calendar time, the adjusted odds ratio is 0.47. The crude estimate is
100% (0.56 – 0.47)/0.56 = 16% higher than the adjusted estimate, therefore calendar
time would be considered a potential confounder.

7.5 Final analyses

After identifying potential confounders, final VE estimates can be calculated.
Generally, VE is estimated using regression models to adjust for covariates. Final
estimates should include variables that are found to be confounders in the study
sample as well as variables chosen a priori. As with any analyses, appropriateness of
the statistical methods should be reviewed with a statistical expert with respect to
model diagnostics and validity checks (e.g. goodness of fit, identification of outliers,
and assessment for multiple co-linearity).

7.6 Pooling data from multiple VE studies

As noted in the sample size calculations, large sample sizes are needed to estimate VE
against specific influenza types/subtypes or to estimate VE separately for different
vaccine products or age groups. One approach to enable stratified estimates by age,
vaccine product, or by virus type/subtype is to pool data from a single study setting
across multiple influenza seasons. This approach has been used by the US Flu VE
Network, for example, to estimate the relative effectiveness of live attenuated vaccines
vs inactivated vaccines in young children. 26

An alternate approach is to pool data from separate VE studies conducted in different

countries within a region. This can be through meta-analysis of reported VE estimates,
or through the pooling of individual level subject data. In either case, pooling data has
several challenges that must be considered:

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 29
i)  he studies being pooled need to be measuring the same vaccine effect. The
T
protocols must be sufficiently similar in terms of case definitions, exclusionary
criteria, sampling timeframes, and vaccinations under study to make pooling
meaningful. For example, it would not be meaningful to pool data from a
hospital-based VE study with data from an outpatient-based study, as VE against
influenza hospitalizations is unlikely to be the same as VE against outpatient
illness.

ii)  he studies to be pooled must have similar data available on key covariates to
T
include in adjusted VE models.

iii) P
 erhaps most importantly, the study settings must be similar enough that the
pooled results can be generalized to each study setting. This means that the
populations under study must have comparable access to vaccination and to
health care for acute respiratory illness. Influenza viruses may differ antigenically
between different study sites, particularly when they are in different hemispheres
or climate zones.

For these reasons, WHO cautions against pooling data from populations that are
heterogeneous with respect to: influenza vaccine products, programmes or policies;
health systems or care-seeking behaviours; or influenza infection risk overall or by
type/subtype. This would apply to pooling data from special populations, such as
those in prisons or nursing homes, with general community-dwelling populations.
If pooled analyses are to be attempted, standard practices should be followed with
regard to analysis of heterogeneity between populations.

30 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
 8. Building a test-negative
study onto SARI surveillance

One attractive approach to estimating VE in low-resource settings where influenza

vaccine has been widely distributed is to build a test-negative study onto an existing
sentinel hospital surveillance system. In this approach, patients hospitalized for ARI
would be identified at surveillance hospitals as part of the existing SARI surveillance
programme.92 Respiratory specimens (nasopharyngeal or oropharyngeal swabs or
nasopharyngeal washes) would be collected from all (or a random sample of) the
SARI patients. These specimens would be tested for evidence of influenza virus,
preferably using RT-PCR. Study staff or surveillance staff would interview patients,
their guardians, or other family members to assess influenza vaccination history,
the course of illness, and important potential confounders. Vaccination history
could be validated through medical record review, vaccine registry data, or other
documentation. VE would then be estimated as (1 – OR), where OR is the ratio of the
odds of vaccination among SARI patients testing positive for influenza to the odds of
vaccination in patients testing negative, adjusted for confounding factors.

This approach has the following advantages:

i) It has severe disease as an outcome, which is of interest to policy-makers.

ii)  ll cases are laboratory-confirmed, which makes it easier to detect a vaccine

A
effect on disease and reduces confounding relative to studies that do not use
laboratory-confirmed outcomes.

iii) W
 hen SARI surveillance has already been established, most of the infrastructure
needed for this type of study is already in place, although efforts to verify
vaccination history from records or registries would need to be added.

iv) T
 he test-negative design reduces the strength of confounding by differences
in health-care seeking between vaccinated and unvaccinated persons and by
community variation in vaccine coverage.

Although this approach to VE studies has several advantages, it is important to

stress that the use the test-negative design has been much less used in hospital-based
studies than in outpatient-based studies. It is even less studied in hospital-based
settings in low- or middle-income countries where the patient base may differ from

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 31
that in more developed settings. Consequently, important sources of bias may not
yet be well understood. Below are some considerations for conducting test-negative
studies within SARI surveillance systems. It is highly advisable that the full study be
conducted by scientists of sufficient epidemiologic expertise to address the complex
issues discussed in this document during design and implementation of the study, and
that a full study be preceded by pilot testing of methods for data collection, validating
vaccination history, case identification, and laboratory testing.

8.1 SARI case definition

While WHO has proposed a SARI case definition,39 the SARI definition used in
practice may differ.54,93 Several aspects of the SARI case definition are important to
consider if SARI surveillance is to be used for a test-negative study. SARI patients
need to be identified soon enough that influenza testing remains highly sensitive. If
subjects are enrolled after they are no longer shedding influenza virus, laboratory
testing will falsely classify them as influenza-negative. These patients will be classified
as non-cases in the analysis, biasing VE estimates downwards. Most existing test-
negative studies restrict eligibility to patients whose onset was within the past seven
days. 20,94 To reduce the risk of false negative tests, SARI surveillance systems that use
the WHO-recommended 10-day post-onset window should consider excluding from
VE studies SARI patients with onset prior to the past seven days.52

The specificity of the case definition also needs to be considered. A broad list of
inclusionary signs and symptoms may increase the probability that the case definition
will capture persons hospitalized due to influenza. However, a broad case definition
also increases the contribution of other conditions to the SARI patient population.
Test-negative studies of hospitalized patients that use broad case definitions can
produce substantial differences in the prevalence of chronic conditions between cases
and non-cases.32 This can introduce confounding by underlying health status.58 If a
test-negative study is based on SARI surveillance that does not use the WHO SARI
case definition, the study participants should be restricted to SARI patients who have
evidence of an acute infection and not some underlying chronic cause.

8.2 Laboratory testing

If a SARI surveillance platform is to be used for a test-negative VE study, another
important factor is who in the surveillance system decides to test specimens from
patients for influenza virus. Is laboratory testing based on a physician’s clinical
decision-making, or is it based on the use of case definitions by surveillance staff?
Selection bias (discussed in Section 5) may be introduced if individual clinicians
determine which patients are tested, as vaccinated patients may be less likely to
have specimens tested than unvaccinated patients. Even if physicians’ decisions are
not based explicitly on vaccine history, clinical testing decisions are based on many
factors that could exert subtle biases in a test-negative study.

Whether or not clinical testing results in a population sample that is biased with
respect to vaccination history, clinicians are likely to preferentially order testing of
specimens from particular subgroups of the population (such as young children).
Relying on clinical testing would skew the VE study participants towards these

32 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
groups, which may or may not be the groups for which VE estimates are of highest
interest. To avoid biases and sub-optimal populations for testing, the decisions on
testing for SARI surveillance should be made by surveillance personnel and based on
pre-defined case definitions.

8.3 Timing of enrolment

The seasonality of influenza varies considerably around the world, ranging from
well-defined seasonal epidemics to year-round circulation.11 Recruitment of subjects
from SARI surveillance into a test-negative study should only be done during
periods when influenza viruses are circulating in the population under surveillance.
If subjects are recruited outside of an influenza season, they will almost invariably
test negative for influenza and be treated as non-cases. Including non-cases from time
periods when influenza is not circulating can lead to biased VE estimates, if influenza
vaccine coverage varies between influenza epidemics and other time periods.79 For
example, in high-resource temperate settings, influenza vaccine is usually distributed
in the autumn, prior to the winter influenza epidemic. If subjects are enrolled into a
test-negative study in the autumn, nearly all will be non-cases. Because vaccine is still
being distributed, vaccine coverage is lower in the autumn than in the winter. Thus,
inclusion of these non-cases will lead to underestimation of the vaccine coverage
relative to coverage during influenza season. In settings where influenza seasons are
less well-defined, or where influenza may circulate year-round, then adjusting for
calendar time (perhaps with indicator variables for calendar month) will be needed
to control for differences in vaccine coverage over time.79 But when influenza has a
defined seasonality, adjusting for calendar time will mean that non-cases enrolled
outside the influenza season will have few if any cases in their time strata, and will
thus contribute little to the analysis, wasting resources used to recruit these subjects.

8.4 Other considerations

As with any observational study design, VE estimates from a test-negative study can
be biased if vaccinated and unvaccinated people differ in the rate at which they come
in contact with individuals infected with influenza virus.79 In a test-negative design,
this bias could be manifested if there is a localized influenza outbreak among a
particular subset of the population, such as prison inmates or health-care workers at a
specific facility. Vaccine coverage among these influenza cases is likely to be different
than vaccine coverage in the general population, from which the test-negative subjects
are drawn, leading to biased VE estimates. When conducting a test-negative study,
researchers should be aware of localized outbreaks and assess whether to include
patients from these outbreaks in the test-negative study. 

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 33
 9. Reporting influenza
VE results

As with all observational studies, reports of influenza VE studies should include

sufficient details on the study participants, data collection, and analyses to enable readers
to judge the validity of the study. The Strengthening the Reporting of Observational
Studies in Epidemiology (STROBE) statement provides recommendations for a
minimum set of reporting elements.95 These include descriptions of setting, dates of
enrolment and follow-up, case definitions and exposure measurement, sample sizes
and patients included/excluded, and key characteristics of the study participants. The
STROBE guidelines provide a starting point for VE reporting. However, influenza
vaccines are updated every year, with the aim of matching the continually changing
viral strains, the relative prevalence of which varies geographically within a single
season. Reports of influenza VE studies need to include additional information to
allow readers to assess whether the study results can be generalized beyond the
specific season and setting under study. WHO encourages investigators to share their
raw individual-level, anonymised data from VE studies.80 This would greatly aid
pooling of results.

9.1 Description of influenza vaccines

To evaluate VE estimates, it is necessary to know which influenza vaccines and
specific populations are being studied. At a minimum, this requires listing the
virus strains that are included in the vaccine; for example, the northern hemisphere
2015/16 trivalent vaccine contained A/California/7/2009(H1N1)pdm-like virus,
A/Switzerland/9715293/2013(A(H3N2)-like virus, and B/Phuket/3073/2013-like
virus.16 In settings where a limited number of vaccine products are in use, the specific
products should also be described, listing the manufacturer and product. Even in
settings with a variety of vaccine products in use, efforts should be made to estimate
the coverage with different vaccine types (live-attenuated vs inactivated, trivalent vs
quadrivalent, standard dose vs high dose) among the study participants. It is generally
difficult to enrol sufficient subjects to permit estimates of vaccine product-specific
VE. However, including data on the relative frequencies of different vaccine products
aids the comparison of VE estimates with other studies.

34 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
9.2 Description of circulating influenza viruses
It is also necessary to know the characteristics of influenza viruses that circulated in
the periods during which VE was estimated. At a minimum, this includes describing
the proportion of study subjects infected with A (H1N1), A (H3N2), B Yamagata,
and B Victoria viruses. Therefore, in study settings where molecular characterization
cannot be done, efforts should be made to transport influenza-positive samples to
reference laboratories that can perform this testing. Because VE can vary by type,
subtype, and lineage, these data allow comparison of VE estimates with VE estimates
in other countries during the same time period. Ideally, these data should be coupled
with antigenic characterization data from at least a subset of the viruses, as differences
in VE across countries or settings could be due to the circulating of antigenically
distinct viruses.

9.3 Stratification of VE estimates

An overall estimate of influenza VE against all viruses in the total population under
study provides data on how well the vaccine worked in that setting, for that year.
Generalizing the VE estimate to other years or settings requires a more thorough
understanding of the season and its composition per type/subtype. This is done
through stratifying VE estimates by virus and by age group. Unless circulating
influenza is highly dominated by a single subtype or lineage,96 VE estimates should be
stratified by virus type, subtype, and lineage, assuming the sample size is sufficient.
In settings where vaccine is distributed to multiple age groups, WHO recommends
that VE estimates be stratified by age, as VE can vary across age groups. Common
age groups for stratified VE estimates include children <9 years of age (for whom two
doses may be needed to generate immunity when vaccinated against influenza for the
first time), adults 18-49 years of age (for whom the highest VE is expected), and adults
65 years of age and older (for whom VE may sometimes be lower relative to other age
groups).

In some settings it is possible to further characterize contributing influenza viruses

genomically and antigenically. 21,97 This enables interpretation of VE in relation to
vaccine match/mismatch. In some instances, it may be possible to calculate VE
specifically against viruses that are clade-level or antigenically matched/mismatched
to vaccine. This is particularly relevant as laboratory proxies of vaccine match/
mismatch to circulating influenza strains may be poor surrogates for anticipating
actual vaccine protection. Linking genetic, antigenic and epidemiologic results may
be informative in the short-term for explaining suboptimal vaccine performance
during some seasons, in the intermediate term for guiding annual reformulation of
vaccine, and in the long term for improving the understanding of genetic and antigenic
markers of virus change and determinants/predictors of vaccine performance. It
is recognized, however, that not all centres may have the capacity to directly link
detailed laboratory findings to epidemiologic measures of vaccine performance and
this may not be a priority undertaking in low-resource settings.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 35
 References

1.  ozano R, Naghavi M, Foreman K, et al. Global and regional mortality from

L
235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for
the Global Burden of Disease Study 2010. Lancet. 2012;380(9859):2095-2128.

2. N
 air H, Brooks WA, Katz M, et al. Global burden of respiratory infections due
to seasonal influenza in young children: a systematic review and meta-analysis.
Lancet. 2011;378(9807):1917-1930.

3. World Health Organization. Vaccines against influenza WHO position paper

- November 2012. Wkly Epidemiol Rec. 2012;87(47):461-476.

4.  orld Health Organization. Generic protocol for monitoring impact of

W
rotavirus vaccination on gastroenteritis disease burden and viral strains. http://
apps.who.int/iris/bitstream/10665/69913/1/WHO_IVB_08.16_eng.pdf.

5. W
 orld Health Organization. Principles and considerations for adding a vaccine
to a national immunization programme: From decision to implementation and
monitoring. 2014; http://www.who.int/immunization/programmes_systems/
policies_strategies/vaccine_intro_resources/nvi_guidelines/en/. Accessed 7
March 2017.

6. World Health Organization. Measuring impact of Streptococcus pneumoniae

and Haemophilus influenzae type b conjugate vaccination. http://apps.who.int/
iris/bitstream/10665/75835/1/WHO_IVB_12.08_eng.pdf. Accessed 7 March 2017.

7.  l Omeiri N, Azziz-Baumgartner E, Clara W, et al. Pilot to evaluate the

E
feasibility of measuring seasonal influenza vaccine effectiveness using surveillance
platforms in Central-America, 2012. BMC public health. 2015;15(1):673.

8. Kissling E, Valenciano M, Buchholz U, et al. Influenza vaccine effectiveness

estimates in Europe in a season with three influenza type/subtypes circulating:
the I-MOVE multicentre case-control study, influenza season 2012/13.
Euro surveillance : bulletin Europeen sur les maladies transmissibles = European
communicable disease bulletin. 2014;19(6).

36 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
9.  cLean HQ, Thompson MG, Sundaram ME, et al. Influenza vaccine
M
effectiveness in the United States during 2012-2013: variable protection by age
and virus type. The Journal of infectious diseases. 2015;211(10):1529-1540.

10. C
 ox NJ, Subbarao K. Influenza. Lancet. 1999;354(9186):1277-1282.

11. Moura FE. Influenza in the tropics. Current opinion in infectious diseases.
2010;23(5):415-420.

12. Morens DM, Fauci AS. The 1918 influenza pandemic: insights for the 21st
century. The Journal of infectious diseases. 2007;195(7):1018-1028.

13. Cowling BJ, Perera RA, Fang VJ, et al. Incidence of influenza virus infections
in children in Hong Kong in a 3-year randomized placebo-controlled vaccine
study, 2009-2012. Clinical infectious diseases : an official publication of the
Infectious Diseases Society of America. 2014;59(4):517-524.

14. K
 uster SP, Shah PS, Coleman BL, et al. Incidence of influenza in healthy
adults and healthcare workers: a systematic review and meta-analysis. PloS one.
2011;6(10):e26239.

15. Omer SB, Goodman D, Steinhoff MC, et al. Maternal influenza immunization
and reduced likelihood of prematurity and small for gestational age births: a
retrospective cohort study. PLoS medicine. 2011;8(5):e1000441.

16. World Health Organization. Recommended composition of influenza virus

vaccines for use in the 2015-2016 northern hemisphere influenza season. Wkly
Epidemiol Rec. 2015;90(11):97-108.

17. M
 embers of the Western Pacific Region Global Influenza Surveillance
Response System, Dwyer D, Barr I, et al. Seasonal influenza vaccine policies,
recommendations and use in the World Health Organization’s Western Pacific
Region. Western Pacific surveillance and response journal : WPSAR. 2013;4(3):51-59.

18. DiazGranados CA, Dunning AJ, Kimmel M, et al. Efficacy of high-dose versus
standard-dose influenza vaccine in older adults. The New England journal of
medicine. 2014;371(7):635-645.

19. B
 elongia EA, Kieke BA, Donahue JG, et al. Effectiveness of inactivated influenza
vaccines varied substantially with antigenic match from the 2004-2005 season to
the 2006-2007 season. J Infect Dis. 2009;199(2):159-167.

20. Ohmit SE, Thompson MG, Petrie JG, et al. Influenza Vaccine Effectiveness
in the 2011-2012 Season: Protection Against Each Circulating Virus and the
Effect of Prior Vaccination on Estimates. Clinical infectious diseases : an official
publication of the Infectious Diseases Society of America. 2013;58(3):319-327.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 37
21.  kowronski DM, Chambers C, Sabaiduc S, et al. Integrated Sentinel Surveillance
S
Linking Genetic, Antigenic, and Epidemiologic Monitoring of Influenza
Vaccine-Virus Relatedness and Effectiveness During the 2013-2014 Influenza
Season. The Journal of infectious diseases. 2015;212(5):726-739.

22. Sullivan SG, Feng S, Cowling BJ. Potential of the test-negative design for
measuring influenza vaccine effectiveness: a systematic review. Expert review of
vaccines. 2014;13(12):1571-1591.

23. T
regnaghi M, Ceballos A, Ruttimann R, et al. Active epidemiologic
surveillance of pneumonia and invasive pneumococcal disease in ambulatory
and hospitalized infants in Cordoba, Argentina. The Pediatric infectious disease
journal. 2006;25(4):370-372.

24. L
 ucero MG, Williams G. Vaccine trial as “probe” to define the burden of
pneumococcal pneumonia disease. Lancet. 2005;365(9465):1113-1114.

25. Gessner BD, Brooks WA, Neuzil KM, et al. Vaccines as a tool to estimate
the burden of severe influenza in children of low-resourced areas (November
30-December 1, 2012, Les Pensieres, Veyrier-du-Lac, France). Vaccine.
2013;31(32):3222-3228.

26. Centers for Disease Control and Prevention. ACIP votes down use of LAIV
for 2016-2017 flu season. 2016; http://www.cdc.gov/media/releases/2016/s0622-
laiv-flu.html. Accessed 7 March 2017.

27. B
 ecker-Dreps S, Amaya E, Liu L, et al. Changes in childhood pneumonia
and infant mortality rates following introduction of the 13-valent pneumococcal
conjugate vaccine in Nicaragua. The Pediatric infectious disease journal.
2014;33(6):637-642.

28. Pirez MC, Algorta G, Chamorro F, et al. Changes in hospitalizations for

pneumonia after universal vaccination with pneumococcal conjugate vaccines
7/13 valent and haemophilus influenzae type b conjugate vaccine in a Pediatric
Referral Hospital in Uruguay. The Pediatric infectious disease journal.
2014;33(7):753-759.

29. Black S, Shinefield H, Fireman B, et al. Efficacy, safety and immunogenicity of

heptavalent pneumococcal conjugate vaccine in children. Northern California
Kaiser Permanente Vaccine Study Center Group. The Pediatric infectious disease
journal. 2000;19(3):187-195.

30. C
 utts FT, Zaman SM, Enwere G, et al. Efficacy of nine-valent pneumococcal
conjugate vaccine against pneumonia and invasive pneumococcal disease in
The Gambia: randomised, double-blind, placebo-controlled trial. Lancet.
2005;365(9465):1139-1146.

38 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
31. Belongia EA, Kieke BA, Donahue JG, et al. Effectiveness of inactivated influenza
vaccines varied substantially with antigenic match from the 2004-2005 season to
the 2006-2007 season. The Journal of infectious diseases. 2009;199(2):159-167.

32. Kwong JC, Campitelli MA, Gubbay JB, et al. Vaccine effectiveness against
laboratory-confirmed influenza hospitalizations among elderly adults during
the 2010-2011 season. Clinical infectious diseases : an official publication of the
Infectious Diseases Society of America. 2013;57(6):820-827.

33. Osterholm MT, Kelley NS, Sommer A, Belongia EA. Efficacy and effectiveness
of influenza vaccines: a systematic review and meta-analysis. Lancet Infect Dis.
2012;12(1):36-44.

34. K
 issling E, Nunes B, Robertson C, et al. I-MOVE multicentre case-control
study 2010/11 to 2014/15: Is there within-season waning of influenza type/
subtype vaccine effectiveness with increasing time since vaccination? Euro
surveillance : bulletin Europeen sur les maladies transmissibles = European
communicable disease bulletin. 2016;21(16).

35. M
 cIntyre AF, Gonzalez-Feliciano AG, Bryan LN, et al. Seasonal influenza
vaccination coverage - United States, 2009-10 and 2010-11. Morbidity and
mortality weekly report Surveillance summaries. 2013;62 Suppl 3:65-68.

36. Loerbroks A, Stock C, Bosch JA, Litaker DG, Apfelbacher CJ. Influenza
vaccination coverage among high-risk groups in 11 European countries.
European journal of public health. 2012;22(4):562-568.

37. C
 enters for Disease Control and Prevention. Flu Vaccination Coverage:
United States, 2015-16 Influenza Season 2016; https://www.cdc.gov/flu/pdf/
fluvaxview/2015-16/nfid-coverage-2015-16-final.pdf. Accessed 3 February
2017.

38. World Health Organization. WHO Manual for estimating the economic burden
of seasonal influenza. 2016; http://apps.who.int/iris/bitstream/10665/250085/1/
WHO-IVB-16.04-eng.pdf?ua=1. Accessed 13 December 2016.

39. World Health Organization. Global Epidemiological Surveillance Standards

for Influenza. 2014; http://www.who.int/influenza/resources/documents/
influenza_surveillance_manual/en/. Accessed 7 March 2017.

40. K
 ostova D, Reed C, Finelli L, et al. Influenza Illness and Hospitalizations
Averted by Influenza Vaccination in the United States, 2005-2011. PloS one.
2013;8(6):e66312.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 39
41. D
 ixon GA, Moore HC, Kelly H, et al. Lessons from the first year of the WAIVE
study investigating the protective effect of influenza vaccine against laboratory-
confirmed influenza in hospitalised children aged 6-59 months. Influenza and
other respiratory viruses. 2010;4(4):231-234.

42. M
 acesic N, Kotsimbos TC, Kelly P, Cheng AC. Hospital-acquired influenza
in an Australian sentinel surveillance system. The Medical journal of Australia.
2013;198(7):370-372.

43. R
 eed C, Kim IK, Singleton JA, et al. Estimated influenza illnesses and
hospitalizations averted by vaccination--United States, 2013-14 influenza
season. MMWR Morbidity and mortality weekly report. 2014;63(49):1151-1154.

44. Preaud E, Durand L, Macabeo B, et al. Annual public health and economic
benefits of seasonal influenza vaccination: a European estimate. BMC public
health. 2014;14:813.

45. E
 uropean Medicines Agency. Guideline on Influenza Vaccines. European
Medicines Agency 2016; http://www.ema.europa.eu/docs/en_GB/document_
library/Scientific_guideline/2016/07/WC500211324.pdf. Accessed 3 February
2017.

46. I nfluenza Monitoring of Vaccine Effectiveness (I-MOVE) in Europe. Protocols

/ Reviews. 2016; https://sites.google.com/site/epiflu/docu. Accessed 13 December
2016.

47. E
 uropean Centre for Disease Prevention and Control. Protocol for cohort
database studies to measure influenza vaccine effectiveness in the European
Union and European Economic Area Member States 2009; http://ecdc.europa.
eu/en/publications/Publications/0907_TER_ Influenza_ AH1N1_ Measuring_
Influenza_Vaccine_ Effectiveness_ Protocol_Cohort_ Database_ Studies.pdf
Accessed 10 March 2017.

48. E
 uropean Centre for Disease Prevention and Control. Protocol for case-
control studies to measure influenza vaccine effectiveness in the European
Union and European Economic Area Member States. 2009; http://ecdc.europa.
eu/en/publications/Publications/0907_TED_ Influenza_ AH1N1_ Measuring_
Influenza_Vaccine_ Effectiveness_ Protocol_ Case_ Control_ Studies.pdf.
Accessed 10 March 2017.

49. E
 uropean Centre for Disease Prevention and Control. Protocol for cluster
investigations to measure influenza vaccine effectiveness in the EU/EEA. 2009;
http://ecdc.europa.eu/en/publications/Publications/0912_TED_ Protocol_for_
Cluster_Investigations_to_Measure_Influenza_Vaccine_Effectiveness.pdf.

40 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
50.  ilca R, Amini R, Douville-Fradet M, et al. Other respiratory viruses are
G
important contributors to adult respiratory hospitalizations and mortality even
during peak weeks of the influenza season. Open forum infectious diseases.
2014;1(2):ofu086.

51. Babcock HM, Merz LR, Dubberke ER, Fraser VJ. Case-control study of clinical
features of influenza in hospitalized patients. Infection control and hospital
epidemiology : the official journal of the Society of Hospital Epidemiologists of
America. 2008;29(10):921-926.

52. Murray EL, Khagayi S, Ope M, et al. What are the most sensitive and specific
sign and symptom combinations for influenza in patients hospitalized with
acute respiratory illness? Results from western Kenya, January 2007-July 2010.
Epidemiology and infection. 2013;141(1):212-222.

53. Guan XR, Li X, Xin XM, et al. Influenza virus infection and risk of acute
myocardial infarction. Inflammation. 2008;31(4):266-272.

54. V
 erani JR, McCracken J, Arvelo W, et al. Surveillance for hospitalized acute
respiratory infection in Guatemala. PloS one. 2013;8(12):e83600.

55. Simmerman JM, Uyeki TM. The burden of influenza in East and South-East
Asia: a review of the English language literature. Influenza and other respiratory
viruses. 2008;2(3):81-92.

56. F
 erdinands JM, Gargiullo P, Haber M, Moore M, Belongia EA, Shay
DK. Inactivated influenza vaccines for prevention of community-acquired
pneumonia: the limits of using nonspecific outcomes in vaccine effectiveness
studies. Epidemiology. 2013;24(4):530-537.

57. Tallo VL, Kamigaki T, Tan AG, et al. Estimating influenza outpatients’ and
inpatients’ incidences from 2009 to 2011 in a tropical urban setting in the
Philippines. Influenza and other respiratory viruses. 2014;8(2):159-168.

58. Jackson ML, Nelson JC, Weiss NS, Neuzil KM, Barlow W, Jackson LA.
Influenza vaccination and risk of community-acquired pneumonia in
immunocompetent elderly people: a population-based, nested case-control
study. Lancet. 2008;372(9636):398-405.

59. Fedson DS, Wajda A, Nicol JP, Hammond GW, Kaiser DL, Roos LL. Clinical
effectiveness of influenza vaccination in Manitoba. JAMA. 1993;270(16):1956-
1961.

60. N
 ichol KL, Nordin JD, Nelson DB, Mullooly JP, Hak E. Effectiveness of
influenza vaccine in the community-dwelling elderly. The New England journal
of medicine. 2007;357(14):1373-1381.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 41
61. B
 axter R, Lee J, Fireman B. Evidence of bias in studies of influenza vaccine
effectiveness in elderly patients. The Journal of infectious diseases. 2010;201(2):186-
189.

62. Jackson LA, Jackson ML, Nelson JC, Neuzil KM, Weiss NS. Evidence of
bias in estimates of influenza vaccine effectiveness in seniors. Int J Epidemiol.
2006;35(2):337-344.

63. Campitelli MA, Rosella LC, Stukel TA, Kwong JC. Influenza vaccination and
all-cause mortality in community-dwelling elderly in Ontario, Canada, a cohort
study. Vaccine. 2010;29(2):240-246.

64. Savitz DA, Fell DB, Ortiz JR, Bhat N. Does influenza vaccination improve
pregnancy outcome? Methodological issues and research needs. Vaccine.
2015;33(47):6430-6435.

65. Remschmidt C, Wichmann O, Harder T. Frequency and impact of confounding

by indication and healthy vaccinee bias in observational studies assessing influenza
vaccine effectiveness: a systematic review. BMC infectious diseases. 2015;15:429.

66. Jackson ML, Rothman KJ. Effects of imperfect test sensitivity and specificity on
observational studies of influenza vaccine effectiveness. Vaccine. 2015;33(11):1313-
1316.

67. H
 avers F, Thaker S, Clippard JR, et al. Use of influenza antiviral agents by
ambulatory care clinicians during the 2012-2013 influenza season. Clinical
infectious diseases : an official publication of the Infectious Diseases Society of
America. 2014;59(6):774-782.

68.  cNeil S, Shinde V, Andrew M, et al. Interim estimates of 2013/14 influenza

M
clinical severity and vaccine effectiveness in the prevention of laboratory-
confirmed influenza-related hospitalisation, Canada, February 2014. Euro
surveillance : bulletin Europeen sur les maladies transmissibles = European
communicable disease bulletin. 2014;19(9).

69. Fu C, He Q, Li Z, et al. Seasonal influenza vaccine effectiveness among children,

2010-2012. Influenza and other respiratory viruses. 2013;7(6):1168-1174.

70. Jackson ML, Henrikson NB, Grossman DC. Evaluating Washington State’s
immunization information system as a research tool. Academic pediatrics.
2014;14(1):71-76.

71. Brown C, Clayton-Boswell H, Chaves SS, et al. Validity of parental report

of influenza vaccination in young children seeking medical care. Vaccine.
2011;29(51):9488-9492.

42 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
72. I rving SA, Donahue JG, Shay DK, Ellis-Coyle TL, Belongia EA. Evaluation
of self-reported and registry-based influenza vaccination status in a Wisconsin
cohort. Vaccine. 2009;27(47):6546-6549.

73. Jimenez-Garcia R, Hernandez-Barrera V, Rodriguez-Rieiro C, et al.

Comparison of self-report influenza vaccination coverage with data from a
population based computerized vaccination registry and factors associated with
discordance. Vaccine. 2014;32(35):4386-4392.

74. M
 angtani P, Shah A, Roberts JA. Validation of influenza and pneumococcal
vaccine status in adults based on self-report. Epidemiology and infection.
2007;135(1):139-143.

75. Z
 immerman RK, Raymund M, Janosky JE, Nowalk MP, Fine MJ. Sensitivity and
specificity of patient self-report of influenza and pneumococcal polysaccharide
vaccinations among elderly outpatients in diverse patient care strata. Vaccine.
2003;21(13-14):1486-1491.

76. Fireman B, Lee J, Lewis N, Bembom O, van der Laan M, Baxter R. Influenza
vaccination and mortality: differentiating vaccine effects from bias. Am J
Epidemiol. 2009;170(5):650-656.

77. J ackson LA, Nelson JC, Benson P, et al. Functional status is a confounder of
the association of influenza vaccine and risk of all cause mortality in seniors. Int
J Epidemiol. 2006;35(2):345-352.

78. S
 imonsen L, Taylor RJ, Viboud C, Miller MA, Jackson LA. Mortality benefits
of influenza vaccination in elderly people: an ongoing controversy. Lancet Infect
Dis. 2007;7(10):658-666.

79. J ackson ML, Nelson JC. The test-negative design for estimating influenza
vaccine effectiveness. Vaccine. 2013;31(17):2165-2168.

80. Bond HS, Sullivan SG, Cowling BJ. Regression approaches in the test-negative
study design for assessment of influenza vaccine effectiveness. Epidemiology
and infection. 2016;144(8):1601-1611.

81. Jacoby P, Kelly H. Is it necessary to adjust for calendar time in a test negative
design?: Responding to: Jackson ML, Nelson JC. The test negative design for
estimating influenza vaccine effectiveness. Vaccine 2013;31(April (17)):2165-8.
Vaccine. 2014;32(25):2942.

82. Hudgens MG, Halloran ME. Causal Vaccine Effects on Binary Postinfection
Outcomes. J Am Stat Assoc. 2006;101(473):51-64.

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 43
83. Njuguna H, Ahmed J, Oria PA, et al. Uptake and effectiveness of monovalent
influenza A (H1N1) pandemic 2009 vaccine among healthcare personnel in
Kenya, 2010. Vaccine. 2013;31(41):4662-4667.

84. Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design
for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31(30):3104-
3109.

85. De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design:
validity, accuracy and precision of vaccine efficacy estimates compared to the
gold standard of randomised placebo-controlled clinical trials. Euro surveillance
: bulletin Europeen sur les maladies transmissibles = European communicable
disease bulletin. 2013;18(37).

86. F
 arrington CP. Estimation of vaccine effectiveness using the screening method.
Int J Epidemiol. 1993;22(4):742-746.

87. W
 orld Health Organization. Global routine vaccination coverage,2011. Wkly
Epidemiol Rec. 2012;87(44):432-435.

88.  ielding JE, Grant KA, Tran T, Kelly HA. Moderate influenza vaccine
F
effectiveness in Victoria, Australia, 2011. Euro surveillance : bulletin Europeen
sur les maladies transmissibles = European communicable disease bulletin.
2012;17(11).

89. Hernan MA, Hernandez-Diaz S, Werler MM, Mitchell AA. Causal knowledge
as a prerequisite for confounding evaluation: an application to birth defects
epidemiology. Am J Epidemiol. 2002;155(2):176-184.

90. Greenland S. Modeling and variable selection in epidemiologic analysis. Am J

Public Health. 1989;79(3):340-349.

91. Mickey RM, Greenland S. The impact of confounder selection criteria on effect
estimation. Am J Epidemiol. 1989;129(1):125-137.

92. Lafond KE, Tam JS, Bresee JS, Widdowson MA. International meeting on
influenza vaccine effectiveness, 3-4 December 2012, Geneva, Switzerland.
Vaccine. 2014;32(49):6591-6595.

93.  asreen S, Luby SP, Brooks WA, et al. Population-based incidence of severe
N
acute respiratory virus infections among children aged <5 years in rural
Bangladesh, June-October 2010. PloS one. 2014;9(2):e89978.

44 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
94. S
 kowronski D, Chambers C, Sabaiduc S, et al. Interim estimates of 2013/14
vaccine effectiveness against influenza A(H1N1)pdm09 from Canada s sentinel
surveillance network, January 2014. Euro surveillance : bulletin Europeen sur les
maladies transmissibles = European communicable disease bulletin. 2014;19(5):4-17.

95. v
 on Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting
of Observational Studies in Epidemiology (STROBE) statement: guidelines for
reporting observational studies. Lancet. 2007;370(9596):1453-1457.

96. Flannery B, Thaker SN, Clippard J, et al. Interim estimates of 2013-14

seasonal influenza vaccine effectiveness - United States, February 2014. MMWR
Morbidity and mortality weekly report. 2014;63(7):137-142.

97.  kowronski DM, Chambers C, Sabaiduc S, et al. Interim estimates of 2014/15

S
vaccine effectiveness against influenza A(H3N2) from Canada’s Sentinel
Physician Surveillance Network, January 2015. Euro surveillance : bulletin
Europeen sur les maladies transmissibles = European communicable disease
bulletin. 2015;20(4).

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 45
 Annex 1.
Glossary of terms

Antigenic drift Constantly accumulating mutations in the HA and NA

proteins of influenza A and B viruses which allows them to
escape immune recognition, resulting in repetitive ongoing
epidemic influenza outbreaks

Antigenic shift Major changes in the influenza type A HA antigen, often with
changes also in the NA antigen, caused by reassortment between
different influenza A subtypes, such as between animal and
human subtypes; such resulting viruses can potentially cause
regional outbreaks or a global pandemic

Bias Systematic differences in enrolling or in collecting data on

study subjects based on vaccination history or outcome status

Case-control An observational study design in which subjects with (cases)

study and without (controls) the outcome of interest are identified,
and vaccination history is assessed retrospectively

Cohort study An observational study design in which study subjects are

identified based on vaccination status and followed over time
to identify incident outcomes

Confounding Underlying differences in the risk of study outcomes between

vaccinated and unvaccinated persons that lead to spurious
vaccine effectiveness estimates

Covariates Factors other than outcome and vaccination history that are
measured on all study subjects; examples include age, sex,
and comorbid illnesses; used to produce stratified vaccine
effectiveness estimates and to control for confounding

Influenza-like A clinical syndrome defined by the World Health Organization

illness (ILI) as an acute respiratory infection with measured fever of ≥38 °C,
cough, and onset within the last seven days

Measurement A type of bias, in which subjects’ outcomes, vaccination

error history, or covariates are measured incorrectly

Non-specific A study outcome that is defined from clinical signs and

outcome symptoms without laboratory confirmation of influenza
infection

Outcome A specific, measurable disease or health event; all study subjects

should be at risk for the outcome, and the occurrence or non-
occurrence of the outcome should be known for all subjects

Outcome Incorrectly determining the occurrence of the outcome for

misclassification a study subject; this includes falsely considering a diseased
person to be disease-free, and falsely considering a disease-free
person to be diseased

46 Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies
Potential A covariate that may reduce confounding in VE estimates
confounder when included in a statistical model

Screening A pseudo-ecologic study design, in which individual-level

method vaccination data are available on all cases but only population-
level data on vaccine coverage is available for the source
population

Selection bias Bias that results when the probability of becoming a study
subject differs for vaccinated vs unvaccinated persons (in a
case-control study) or by likelihood of being identified as a
case (in a cohort study)

Severe acute A clinical syndrome defined by the World Health Organization

respiratory as acute respiratory infection with history of fever or
infection (SARI) documented fever of ≥38 °C, cough, onset within the last ten
days, and requiring hospitalization

Source The larger population from which all study subjects are selected
population

Study All persons enrolled in a VE study; outcomes, vaccine history,

participants and covariates should be measured on all study participants

Test-negative A special type of case-control study design, where all study

design subjects meet some testing criteria and are tested for influenza;
cases are those who test positive and comparison subjects are
those who test negative

Vaccine Reduced risk of disease among vaccinated persons attributed

effectiveness (VE) to vaccination in real-world conditions; estimated from
observational studies

Vaccine efficacy Reduced risk of disease among vaccinated persons resulting

from vaccination under ideal circumstances; estimated from
randomized trials

Vaccine impact Reduction in incidence of disease in a population where some

members are vaccinated; usually estimated from ecologic
studies

Evaluation of influenza vaccine effectiveness - A guide to the design and interpretation of observational studies 47
Department of Immunization, Vaccines and Biologicals

World Health Organization

20 Avenue Appia
CH-1211 Geneva 27, Switzerland
vaccines@who.int
http://www.who.int/immunization/en/