DATA COLLECTION
Introduction
• Data Collection Strategies
• Characteristics of Good Measures
• Quantitative and Qualitative Data
• Tools for Collecting Data
2
Data Collection Strategies
• No one best way: decision depends on:
• What you need to know: numbers or stories
• Where the data reside: environment, files, people
• Resources and time available
• Complexity of the data to be collected
• Frequency of data collection
• Intended forms of data analysis
3
Rules for Collecting Data
• Use multiple data collection methods
• Use available data, but need to know
• how the measures were defined
• how the data were collected and cleaned
• the extent of missing data
• how accuracy of the data was ensured
4
Rules for Collecting Data
• If must collect original data:
• be sensitive to burden on others
• pre-test, pre-test, pre-test
• establish procedures and follow them (protocol)
• maintain accurate records of definitions and coding
• verify accuracy of coding, data input
5
Structured Approach
• All data collected in the same way
• Especially important for multi-site and cluster
evaluations so you can compare
• Important when you need to make comparisons
with alternate interventions
6
Use Structured Approach When:
• need to address extent questions
• have a large sample or population
• know what needs to be measured
• need to show results numerically
• need to make comparisons across different sites
or interventions
7
Semi-structured Approach
• Systematic and follow general procedures but data
are not collected in exactly the same way every
time
• More open and fluid
• Does not follow a rigid script
• may ask for more detail
• people can tell what they want in their own way
8
Use Semi-structured Approach
when:
• conducting exploratory work
• seeking understanding, themes, and/or issues
• need narratives or stories
• want in-depth, rich, “backstage” information
• seek to understand results of data that are
unexpected
9
Characteristics of Good Measures
• Is the measure relevant?
• Is the measure credible?
• Is the measure valid?
• Is the measure reliable?
10
Relevance
Does the measure Do not measure
capture what what is easy instead
matters? of what is needed
11
Credibility
Is the measure believable? Will it be
viewed as a reasonable and appropriate
way to capture the information sought?
12
Internal Validity
How well does the Are waiting lists a
measure capture valid measure of
what it is supposed demand?
to?
13
Reliability
A measure’s How reliable are:
precision and • birth weights of
stability- extent to newborn infants?
which the same • speeds measured by
result would be a stopwatch?
obtained with
repeated trials
14
Quantitative Approach
• Data in numerical form
• Data that can be precisely measured
• age, cost, length, height, area, volume, weight, speed,
time, and temperature
• Harder to develop
• Easier to analyze
15
Qualitative Approach
• Data that deal with description
• Data that can be observed or self-reported, but
not always precisely measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and widely
applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports
16
Which Data?
If you: Then Use:
- want to conduct statistical analysis
- want to be precise Quantitative
- know what you want to measure
- want to cover a large group
- want narrative or in-depth information
- are not sure what you are able to measure Qualitative
- do not need to quantify the results
17
Obtrusive vs. Unobtrusive
Methods
Obtrusive Unobtrusive
data collection methods data collection
that directly obtain methods that do not
information from those collect information
being evaluated directly from evaluees
e.g. interviews, surveys, e.g., document analysis,
focus groups GoogleEarth,
observation at a distance
How to Decide on Data Collection
Approach
• Choice depends on the situation
• Each technique is more appropriate in some
situations than others
• Caution: All techniques are subject to bias
Triangulation to Increase
Accuracy of Data
• Triangulation of methods
• collection of same information using different methods
• Triangulation of sources
• collection of same information from a variety of sources
• Triangulation of evaluators
• collection of same information from more than one
evaluator
