Research methods in Business

DATA COLLECTION
Name of Instructor

Name of Instructor
Outline
1 Data Collection Strategies

2 Characteristics of Good Measures

3 Quantitative and Qualitative Data

4 Data collection methods

4 Tools for Collecting Data

2
Data Collection Strategies

No one best way: decision depends on:

❑What you need to know: numbers or stories

❑Where the data reside: environment, files, people
❑Resources and time available
❑Complexity of the data to be collected
❑Frequency of data collection
❑Intended forms of data analysis

3
Data Collection Strategies

Rules for Collecting Data

❑Use multiple data collection methods

❑Use available data, but need to know
➢how the measures were defined
➢how the data were collected and cleaned
➢ the extent of missing data
➢how accuracy of the data was ensured

4
4
Data Collection Strategies

Rules for Collecting Data

If must collect original data:

❖be sensitive to burden on others
❖pre-test, pre-test, pre-test
❖establish procedures and follow them (protocol)
❖maintain accurate records of definitions and coding
❖verify accuracy of coding, data input

5
Data Collection Strategies

Structured Approach

❖All data collected in the same way

❖Especially important for multi-site and cluster evaluations so you can
compare
❖Important when you need to make comparisons with alternate
interventions

6
Data Collection Strategies
Use Structured Approach When:

❖need to address extent questions

❖have a large sample or population
❖know what needs to be measured
❖need to show results numerically
❖need to make comparisons across different sites or interventions

7
Data Collection Strategies
Semi-structured Approach

❖Systematic and follow general procedures but data are not collected in
exactly the same way every time
❖More open and fluid
❖Does not follow a rigid script
➢may ask for more detail
➢people can tell what they want in their own way

8
Data Collection Strategies

Use Semi-structured Approach when:

❖conducting exploratory work

❖seeking understanding, themes, and/or issues
❖need narratives or stories
❖want in-depth, rich, “backstage” information
❖seek to understand results of data that are unexpected

9
Characteristics of Good Measures

➢Is the measure relevant?

➢Is the measure credible?
➢Is the measure valid?
➢Is the measure reliable?

10
Characteristics of Good Measures
Relevance

❖ Does the measure capture what matters?

❖ Do not measure what is easy instead of what

is needed

11
Characteristics of Good Measures

Credibility

❖Is the measure believable?

❖Will it be viewed as a reasonable and appropriate way to capture the
information sought?

12
Characteristics of Good Measures

Internal Validity

❖ How well does the measure capture what it is supposed to?

❖Are waiting lists a valid measure of demand?

13
Characteristics of Good Measures

Reliability

❖ A measure’s precision and stability- extent to which the same

result would be obtained with repeated trials

14
Quantitative and Qualitative Data
Quantitative Approach
• Data in numerical form
• Data that can be precisely measured
• age, cost, length, height, area, volume, weight, speed, time, and
temperature
• Harder to develop
• Easier to analyze

15
15
Quantitative and Qualitative Data
Qualitative Approach
• Data that deal with description
• Data that can be observed or self-reported, but not always precisely
measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and widely applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports

16
16
Quantitative and Qualitative Data
Which Data?

If you: Then Use:

- want to conduct statistical analysis
- want to be precise Quantitative
- know what you want to measure
- want to cover a large group
- want narrative or in-depth information
- are not sure what you are able to measure Qualitative

- do not need to quantify the results

17
17
Data collection methods
Obtrusive vs. Unobtrusive Methods

Obtrusive Unobtrusive
data collection methods data collection
that directly obtain methods that do not
information from those collect information
being evaluated directly from evaluees
e.g. interviews, surveys, e.g., document analysis,
focus groups GoogleEarth,
observation at a distance,
trash of the stars

18
Data collection methods
How to Decide on Data Collection Approach

❑Choice depends on the situation

❑Each technique is more appropriate in some situations than others
❑Caution: All techniques are subject to bias

19
Data collection methods

Triangulation to Increase Accuracy of Data

❖Triangulation of methods
➢collection of same information using different methods
❖Triangulation of sources
➢collection of same information from a variety of sources
❖Triangulation of evaluators
➢collection of same information from more than one evaluator

20
Data Collection Tools

❑Participatory Methods
❑Records and Secondary Data
❑Observation
❑Surveys and Interviews
❑Focus Groups
❑Diaries, Journals, Self-reported Checklists
❑Expert Judgment
❑Delphi Technique
❑Other Tools

21
21
Data Collection Tools
Tool 1: Participatory Methods

❖Involve groups or communities heavily in data collection

❖Examples:
➢community meetings
➢mapping
➢transect walks

22
Data Collection Tools
Tool 1: Participatory Methods

➢ Community Meetings

❑One of the most common participatory methods

❑Must be well organized
➢agree on purpose
➢establish ground rules
✓ who will speak
✓ time allotted for speakers
✓ format for questions and answers

23
Data Collection Tools
Tool 1: Participatory Methods

➢ Mapping

❖Drawing or using existing maps

❖Useful tool to involve stakeholders
❖increases understanding of the community
❖generates discussions, verifies secondary sources of information, perceived
changes
❖Types of mapping:
❖natural resources, social, health, individual or civic assets, wealth, land use,
demographics

24
Data Collection Tools
Tool 1: Participatory Methods

➢ Transect Walks

❖Evaluator walks around community observing people, surroundings, and

resources
❖Need good observation skills
❖Walk a transect line through a map of a community — line should go
through all zones of the community

25
Data Collection Tools
Tool 2: Records and Secondary Data

➢ Examples of sources

❖files/records
❖computer data bases
❖industry or government reports
❖other reports or prior evaluations
❖census data and household survey data
❖electronic mailing lists and discussion groups
❖documents (budgets, organizational charts, policies and procedures,
maps, monitoring reports)
❖newspapers and television reports
26
Data Collection Tools
Tool 2: Records and Secondary Data

➢ Using Existing Data Sets

Key issues: validity, reliability, accuracy, response rates, data

dictionaries, and missing data rates

27
Data Collection Tools
Tool 2: Records and Secondary Data

➢ Advantage/Challenge: Available Data

Advantages Often less expensive and faster than collecting the

original data again
Challenges There may be coding errors or other problems. Data
may not be exactly what is needed. You may have
difficulty getting access. You have to verify validity and
reliability of data

28
Data Collection Tools
Tool 3: Observation

➢ See what is happening

❑traffic patterns
❑land use patterns
❑layout of city and rural areas
❑quality of housing
❑condition of roads
❑conditions of buildings
❑who goes to a health clinic

29
Data Collection Tools
Tool 3: Observation

➢ Observation is Helpful when

❖need direct information

❖trying to understand ongoing behavior
❖there is physical evidence, products, or outputs than can be observed
❖need to provide alternative when other data collection is infeasible or
inappropriate

30
Data Collection Tools
Tool 3: Observation

➢ Degree of Structure of Observations

❖Structured: determine, before the observation, precisely what will be

observed before the observation
❖Unstructured: select the method depending upon the situation with no
pre-conceived ideas or a plan on what to observe
❖Semi-structured: a general idea of what to observe but no specific plan

31
Data Collection Tools
Tool 3: Observation

➢ Ways to Record Information from Observations

❖Observation guide
➢printed form with space to record
❖Recording sheet or checklist
➢Yes/no options; tallies, rating scales
❖Field notes
➢least structured, recorded in narrative, descriptive style

32
Data Collection Tools
Tool 3: Observation

➢ Guidelines for Planning Observations

❖Have more than one observer, if feasible

❖Train observers so they observe the same things
❖Pilot test the observation data collection instrument
❖For less structured approach, have a few key questions in mind

33
Data Collection Tools
Tool 3: Observation

➢ Advantages and Challenges: Observation

Advantages Collects data on actual vs. self- reported behavior or

perceptions. It is real-time vs. retrospective
Challenges Observer bias, potentially unreliable; interpretation and
coding challenges; sampling can be a problem; can be
labor intensive; low response rates

34
Data Collection Tools
Tool 4: Surveys and Interviews

❖Excellent for asking people about:

➢perceptions, opinions, ideas
❖Less accurate for measuring behavior
❖Sample should be representative of the whole
❖Big problem with response rates

35
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Structures for Surveys

❖Structured:
• Precisely worded with a range of pre-determined responses that the
respondent can select
• Everyone asked exactly the same questions in exactly the same
way, given exactly the same choices
❖Semi-structured
• Asks same general set of questions but answers to the questions are
predominantly open-ended

36
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Structured vs. Semi-structured Surveys

Structured harder to develop

easier to complete
easier to analyze
more efficient when working with large numbers
Semi-structured easier to develop: open ended questions
more difficult to complete: burdensome for people to complete as a self-
administrated questionnaire
harder to analyze but provide a richer source of data, interpretation of
open-ended responses subject to bias

37
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Modes of Survey Administration

❖Telephone surveys
❖Self-administered questionnaires distributed by mail, e-mail, or websites
❖Administered questionnaires, common in the development context
❖In development context, often issues of language and translation

38
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Mail / Phone / Internet Surveys

❑Literacy issues
❑Consider accessibility
➢reliability of postal service
➢turn-around time
❑Consider bias
➢What population segment has telephone access? Internet access?

39
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Advantages and Challenges of Surveys

Advantages Best when you want to know what people think,

believe, or perceive, only they can tell you that
Challenges People may not accurately recall their behavior or
may be reluctant to reveal their behavior if it is illegal
or stigmatized. What people think they do or say they
do is not always the same as what they actually do.

40
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Interviews

❖Often semi-structured
❖Used to explore complex issues in depth
❖Forgiving of mistakes: unclear questions can be clarified during the
interview and changed for subsequent interviews
❖Can provide evaluators with an intuitive sense of the situation

41
Data Collection Tools
Tool 4: Surveys and Interviews

➢ Challenges of Interviews

❖Can be expensive, labor intensive, and time consuming

❖Selective hearing on the part of the interviewer may miss information that
does not conform to pre-existing beliefs
❖Cultural sensitivity: e.g., gender issues

42
Data Collection Tools
Tool 5: Focus Groups

❖Type of qualitative research where small homogenous groups of people

are brought together to informally discuss specific topics under the
guidance of a moderator
❖Purpose: to identify issues and themes, not just interesting information,
and not “counts”

43
Data Collection Tools
Tool 5: Focus Groups

➢ Focus Groups Are Inappropriate when

❖language barriers are insurmountable

❖evaluator has little control over the situation
❖trust cannot be established
❖free expression cannot be ensured
❖confidentiality cannot be assured

44
Data Collection Tools
Tool 5: Focus Groups

➢ Focus Group Process

Phase Action

1 Opening Ice-breaker; explain purpose; ground rules; introductions

2 Warm-up Relate experience; stimulate group interaction; start with least threatening
and simplest questions
3 Main body Move to more threatening or sensitive and complex questions; elicit deep
responses; connect emergent data to complex, broad participation
4 Closure End with closure-type questions; summarize and refine; present theories, etc;
invite final comments or insights; thank participants
45
Data Collection Tools
Tool 5: Focus Groups

➢ Advantages and Challenges of Focus Groups

Advantages Can be conducted relatively quickly and easily; may take less staff
time than in-depth, in-person interviews; allow flexibility to make
changes in process and questions; can explore different
perspectives; can be fun
Challenges Analysis is time consuming; participants not be representative of
population, possibly biasing the data; group may be influenced by
moderator or dominant group members

46
46
Data Collection Tools
Tool 6: Diaries and Self-Reported Checklists

➢ Advantages and Challenges of Focus Groups

❑Use when you want to capture information about events in people’s daily
lives
❑Participants capture experiences in real-time not later in a questionnaire
❑Used to supplement other data collection

47
Data Collection Tools
Tool 6: Diaries and Self-Reported Checklists

➢ Guidelines for Diaries or Journals

Step Process
Recruit people face-to-face
1 • encourage participation, appeal to altruism, assure confidentiality, provide
incentive
Provide a booklet to each participant
2 • cover page with clear instructions, definitions, example
• short memory-joggers, explain terms, comments on last page , calendar
Consider the time-period for collecting data
3 • if too long, may become burdensome or tedious
• if too short may miss the behavior or event
48
Data Collection Tools
Tool 6: Diaries and Self-Reported Checklists

➢ Self-reported Checklists

❖Cross between a questionnaire and a diary

❖The evaluator specifies a list of behaviors or events and asks the
respondents to complete the checklist
❖Done over a period of time to capture the event or behavior
❖More quantitative approach than diary

49
Data Collection Tools
Tool 6: Diaries and Self-Reported Checklists

➢ Advantages and Challenges of Diaries and Self-reported Checklists

• Can capture in-depth, detailed data that might be otherwise forgotten

• Can collect data on how people use their time
Advantages • Can collect sensitive information
• Supplements interviews provide richer data

• Requires some literacy

• May change behavior
Challenges • Require commitment and self-discipline
• Data may be incomplete or inaccurate
• Poor handwriting, difficult to understand phrases
50
50
Data Collection Tools
Tool 7: Expert Judgment

❖ Use of experts, one-on-one or as a panel

➢ E.g., Government task forces, Advisory Groups
❖ Can be structured or unstructured
❖ Issues in selecting experts

51
Data Collection Tools
Tool 7: Expert Judgment

➢ Selecting Experts

❖Establish criteria for selecting experts not only on recognition as expert but
also based on:
➢areas of expertise
➢diverse perspectives
➢diverse political views
➢diverse technical expertise

52
Data Collection Tools
Tool 7: Expert Judgment

➢ Advantages and Challenges of Expert Judgment

Advantages Fast, relatively inexpensive

Weak for impact evaluation
May be based mostly on perceptions
Challenges
Value of data depends on how credible the experts are
perceived to be

53
Data Collection Tools
Tool 8: Delphi Technique

❖Enables experts to engage remotely in a dialogue and reach

consensus, often about priorities
❖Experts asked specific questions; often rank choices
❖Responses go to a central source, are summarized and fed back to
the experts without attribution
❖Experts can agree or argue with others’ comments
❖Process may be iterative

54
Data Collection Tools
Tool 8: Delphi Technique

➢ Advantages and Challenges of Delphi Technique

• Allows participants to remain anonymous

• Is inexpensive
• Is free of social pressure, personality influence, and individual
Advantages
dominance
• Is conducive to independent thinking
• Allows sharing of information
• May not be representative
• Has tendency to eliminate extreme positions
Challenges
• Requires skill in written communication
• Requires time and participant commitment
55
Data Collection Tools
Other Measurement Tools

❖scales (weight) ❖health testing tools:

❖tape measure ➢i.e. blood pressure
❖stop watches ❖aptitude and achievement tests
❖chemical tests: ❖citizen report cards
➢i.e. quality of water

56
Data Collection Summary

❖ Choose more than one data collection technique

❖ No “best” tool
❖ Do not let the tool drive your work but rather choose the right tool to
address the evaluation question

57
 A Final Note….
“I never guess. It is a capital mistake
to theorize before one has data.
Insensibly one begins to twist facts and theories,
instead of theories to suit facts.”
--Sir Arthur Conan Doyle

58