Evaluation of Health Programs and Services

Dr\ Adel Aljawfi

Assistant Professor of Family and Community Health

2025

1
 Data Sources in Monitoring and Evaluation

Introduction:

Program evaluation is a systematic way to collect information about the

characteristics, activities, and results of a program in order to make decisions about
the program.
The analytical process and data used for planning provides initial opportunities and
insights to recognize the future monitoring and evaluation requirements in
comparison to existing data sources and quality.
Data Sources:
Data sources are the resources used to obtain data for M&E activities. There are
several levels from which data can come, including client, program, service
environment, population, and geographic levels.
Regardless of level, data are commonly divided into two general categories: routine
and non-routine.
Types of Data Sources:
Routine Data Sources : provide data that are collected on a continuous basis, such
as information that clinics collect on the patients utilizing their services. Although
these data are collected continuously, processing them and reporting on them usually
occur only periodically, for instance, aggregated monthly and reported quarterly.
1. Examples of routine data are:

a. Vital registration records

b. Clinical service statistics

2
 Advantage:

Data collection from routine sources is useful because it can provide information
on a timely basis. For instance, it can be used effectively to detect and correct
problems in service delivery.

Disadvantages:

- It can be difficult to obtain accurate estimates of catchment areas or target

populations through this method, and

- The quality of the data may be poor because of inaccurate record keeping
or incomplete reporting.
Non-routine data sources : provide data that are collected on a periodic basis, usually
annually or less frequently.

1. Examples of non-routine data sources are:

a. Household surveys such as Demographic Health Survey (DHS).

b. National censuses

c. Facility surveys

Advantage:

Nonroutine data can avoid the problem of incorrectly estimating the target population
when calculating coverage indicators. This is particularly the case with representative
population-based surveys, such as a Demographic Health Survey (DHS).

Disadvantages:
- Non-routine data are collecting them is often expensive, and

- This collection is done on an irregular basis. In order to make informed

3
 program decisions, program managers usually need to receive data at more
frequent intervals than non-routine data can accommodate.

- Data from different sources can be used to calculate the same indicator,
although changes to the metric may be necessary. This illustration depicts one
way that routine and non- routine data can be used together to provide for an
effective M&E system.

For example, when calculating the coverage rate for the first dose of a diphtheria-
tetanus- pertussis (DTP) vaccine:

If population-based survey data are used, the definition could be the proportion
of children age 12-23 months who were immunized with the first dose of DTP
vaccine before age 12 months.
 numerator: Number of children age 12-23 months who were
immunized with the first dose of DTP vaccine before age 12 months
 denominator: Total number of children age 12-23 months surveyed

If a routine data source is used, such as service statistics (e.g. clinic records,
outreach records, etc.), the definition could be proportion of infants 0-11 months of
age in a specified calendar year who were immunized with the first dose of DTP
vaccine in that calendar year.
 numerator: Number immunized by age 12 months with the first dose
of DTP vaccine in a given year
 denominator: Total number of surviving infants less than 12 months of age
in the same year.

Data Collection:
The M&E plan should include a data collection plan that summarizes information
about the data sources needed to monitor and/or evaluate the program.

4
The plan should include information for each data source, such as:

1. the timing and frequency of collection;

2. the person or agency responsible for the collection;

3. the information needed for the indicators; and

4. any additional information that will be obtained from the source.

Data Collection Methods:
 Document Review:
Use when: Program documents or literature are available and can provide insight
into the program or the evaluation. You can review meeting minutes, sign in sheets,
quarterly or annual reports, or surveillance data to learn more about the activities of
the program and its reach. You can also review related scientific literature or Web
sites to learn how other similar programs work or what they accomplished.
Advantages:
• Data already exist
• Does not interrupt the program
• Little or no burden on others
• Can provide historical or comparison data
• Introduces little bias
Disadvantages:
• Time consuming
• Data limited to what exists and is available
• Data may be incomplete
 Observation:
Use when: You want to learn how the program actually operates—its processes and
activities. Examples could be observing services being provided, training sessions,

5
 meetings, or special events.
Advantages:
• Allows you to learn about the program as it is occurring
• Can reveal unanticipated information of value
• Flexible in the course of collecting data
Disadvantages:
• Time consuming
• Having an observer can alter events
• Difficult to observe multiple processes simultaneously
• Can be difficult to interpret observed behaviors
 Survey:
Use when: You want information directly from a defined group of people to get a
general idea of a situation, to generalize about a population, or to get a total count of
a particular characteristic. Survey instruments or questionnaires ask questions in a
standardized format that allows consistency and the ability to aggregate responses.
Potential questions can focus on the collection of qualitative or quantitative data.
Advantages:
• Many standardized instruments available
• Allows a large sample
• Standardized responses easy to analyze
• Able to obtain a large amount of data quickly
• Relatively low cost
• Convenient for respondents
Disadvantages:
• Sample may not be representative
• Wording can bias responses
• Closed-ended or brief responses may not provide the “whole story”
6
 • Not suited for all people e.g., those with low reading level.
 Interview:
Used when: You want to understand impressions and experiences in more detail and
be able to expand or clarify responses. Conducting interviews is a method that, like
open-ended questions in a questionnaire, allows you to obtain an individual’s
response in their own words. Interviews differ from questionnaires in that they elicit
more detailed qualitative data and allow you to interact with the person to better
understand their response. Interviews may be conducted in-person or over the phone.
Interviewing is useful when you want more in-depth information about a person’s
attributes, knowledge, attitudes/beliefs, or behaviors.
Advantages:
• Often better response rate than surveys
• Allows flexibility in questions
• Allows more in-depth information to be gathered
Disadvantages:
• Time consuming
• Requires skilled interviewer
• Qualitative data more difficult to analyze.
 Focus Group:
Used when: You want to collect in-depth information from a group of people about
their experiences and perceptions related to a specific issue. Like an interview, a
focus group allows you to collect qualitative data. However, unlike interviews, in
which focus groups provide data about a particular topic through small group
discussions. Focus groups are an excellent method for obtaining opinions about
programs and services.
Advantages:
• Collect multiple peoples’ input in one session
7
 • Allows in-depth discussion
• Group interaction can produce greater insight
• Can be conducted in short time frame
• Can be relatively inexpensive compared to interviews.
Disadvantages:
• Requires skilled facilitator
• Limited number of questions can be asked
• Group setting may inhibit or influence opinions
• Data can be difficult to analyze
• Not appropriate for all topics or populations
Data Quality:
Throughout the data collection process, it is essential that data quality be monitored
and maintained. It is important to consider data quality when determining the
usefulness of various data sources; the data collected are most useful when they are
of the highest quality.

Some types of errors or biases common in data collection include:

1. Sampling bias: occurs when the sample taken to represent the

population of interest is not a representative sample;
2. Non-sampling error: all other kinds of mismeasurement, such as courtesy
bias, incomplete records, incorrect questionnaires, interviewer errors, or
non- response rates; and
3. Subjective measurement: occurs when the data are influenced by the
measurer.
Some data quality issues to consider are:

a. Coverage: Will the data cover all of the elements of interest?

8
 b. Completeness: Is there a complete set of data for each element of interest?

c. Accuracy: Have the instruments been tested to ensure validity and

reliability of the data?
d. Frequency: Are the data collected as frequently as needed?

e. Reporting Schedule: Do the available data reflect the time periods of interest?

f. Accessibility: Are the data needed collectable or retrievable?

g. Power: Is the sample size big enough to provide a stable estimate or

detect change?
Data Use

Collecting data is only meaningful and worthwhile if it is subsequently used for

evidence-based decision-making.

For example, if sales data from a program to provide insecticide-treated bed nets
show that the program is successfully increasing bednet distribution, the decision-
maker may decide to maintain the program. Alternatively, the data may prompt the
implementation of a new distribution system and could spur additional research to
test the effectiveness of this new strategy compared to the existing one. When
decision-makers understand the kinds of information that can be used to inform
decisions and improve results, they are more likely to seek out and use the
information.

,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,