lOMoARcPSD|17741605

RES - 5 - Lecture notes 4

Research 320 (University of Pretoria)

StuDocu is not sponsored or endorsed by any college or university

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)
 lOMoARcPSD|17741605

Chapter 5: Measurement techniques and sampling

methods

• Variable: a condition or characteristic that can take on different values or categories.

Defining measurement
• When we measure, we attempt to identify and characterize the dimensions, quantity, capacity, or
degree of something.
• Measurement: the assignment of symbols or numbers to something according to a set of rules

Scales of measurement
• Measurement can be categorized by the type of information that is communicated by the symbols
assigned to the variables of interest.
• 4 levels of measurement:
1. Nominal scale
2. Ordinal scale
3. Interval scale
4. Ratio scale

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

Nominal scale
• NAME
• It is a nonquantitative scale of measurement because it identifies types rather than amounts
of something.
• Nominal scale: the use of symbols, such as words or numbers, to classify or categorize
measurement objects into groups or types.
• Numbers can be used to label the categories of a nominal variable, but these numbers serve
only as markers, not as indicators of amount or quantity.
• Cats, dogs,
Ordinal scale
• Ordinal scale: a rank order measurement scale.
• Any variable where the levels can be ranked is an ordinal variable.
st nd rd
• 1 , 2 , 3

Interval scale
• Interval scale: a scale of measurement with equal intervals of distance between adjacent numbers.
• No absolute zero.
• Time

Ratio scale
• The highest level of measurement.
• Ratio scale: a scale of measurement with rank ordering, equal intervals, and an absolute zero point.
• It marks/names the values of the variable, provides rank ordering of the values of the variable, and
has equal distances between the values of the variable.
• Temperature

Psychometric properties of good measurement

Overview of reliability and validity
• Reliability refers to the consistency or stability of the scores of your measurement instrument.
• Validity refers to the extent to which your measurement procedure is measuring what you think it
measuring and whether you have used and interpreted the scores correctly.

Reliability
• Reliability: the consistency or stability of scores
• In psychological testing, it refers to the consistency or stability of the scores that we get from a test or
assessment procedure.
• Reliability coefficient: type of correlation coefficient used as an index of reliability
• 4 primary types of reliability:

1. Test-retest reliability
• Test-retest reliability: consistency of a group of individuals’ scores on a test over time.
• To determine the test-retest reliability of a test or research instrument, you administer the test, wait
for a week or so, and administer it again.
• A strong relationship indicates consistency across time.

2. Equivalent forms reliability

• Equivalent forms reliability: consistency of a group of individuals’ scores on two versions of the same
test.

3. Internal consistency reliability

• Internal consistency reliability: consistency with which items on a test measure a single construct.

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

• Internal consistency reliability is affected by the length of the test – as the test gets longer, it becomes
more reliable.
• Estimation of internal consistency only requires one administration of a test or scale, which is
probably the one reason it is commonly reported in journal articles.
• Coefficient alpha: the most frequently used index of internal consistency (coefficient needs to be 0.70
or higher).

4. Interrater reliability
• Interrater reliability: the degree of consistency or agreement between two or more scores, judges, observers, or
raters.
• Interobserver agreement: the percentage of time that different observers’ ratings are in agreement.

Validity
• Validity: accuracy of inferences, interpretations, or actions made on the basis of test scores.
• tests and research instruments always involve the measurement of constructs
• Cosntructs could also characterize a field experimental setting such as an impoverished setting,
enriched setting, or a poverty neighbourhood.
• Operationalization: the way a construct is represented and measured in a particular research study.
• The important issue with respect to validity is whether the operations produce a correct or
appropriate representation of measurement.
• Validation: gathering of evidence regarding the soundness of inferences made from test scores.
• Evidence of validity is obtained by developing a theory about how a test or instrument should operate
if it is working correctly, and then the theory is tested to obtain the evidence.

Validity evidence based on content / Content-related evidence (or content

validity)
• Content – related evidence or content validity: judgement by experts of the degree to which items,
tasks, or questions on a test adequately represent the construct.
• Face validity: prima face judgement of whether the items appear to represent the construct and
whether the test or instrument looks valid.
• When making decisions about content-related evidence, experts collect the necessary data to answer
the following kinds of questions:
o Do the items appear to represent the thing a researcher is attempting to measure? (This
prima facie judgment is sometimes called face validity.)
o Does the set of items underrepresent the construct’s content (i.e., did the researcher
exclude any important content areas or topics)?
o Do any of the items represent something other than what the researcher is trying to
measure (i.e., were any irrelevant items included)?

Validity evidence based on internal structure

• Multidimensional construct: construct consisting of two or more dimensions; contrasted with a
unidimensional construct.
• Factor analysis: a statistical analysis procedure used to determine the number of dimensions present
in a set of items.
• The key point is that the factor analysis results tell you whether the test is unidimensional or
multidimensional.
• Homogeneity: the degree to which a set of items measures a single construct or trait.
Ø 2 primary indices of homogeneity:
1. Item-to-total-correlation ( correlate each item with the total test score)
2. Coefficient Alpha

Validity evidence based on relations to other variables

• This form of evidence is obtained by relating your test scores with one or more relevant and know
criteria.
• A criterion is the standard or benchmark that you want to correlate with or predict accurately on the
basis of your test scores.
• Validity coefficient: the type of correlation coefficient used in validation research

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

• Criterion-related validity: degree to which scores predict or relate to a known criterion such as a
future or an already-established test.
• Two different types of criterion-related validity:
1. Predictive validity: degree to which scores obtained at one time correctly predict the scores
on a criterion at a later time.
2. Concurrent validity: degree to which scores obtained at one time correctly relate to the
scores on a known criterion obtained at approximately the same time
• Convergent validity evidence: validity evidence based on the degree to which the focal test scores
correlate with independent measures of the same construct.
• Discriminant validity evidence: validity evidence based on the degree to which the focal test scores
do not correlate with the measures of different constructs
• Convergent and validity evidence are used together in testing how well a new scale or test operates.
• Known groups validity evidence: degree to which groups that are known to differ on a construct
actually differ according to the test used to measure the construct.

Using reliability and validity information

• Norming group: the reference group upon which reported reliability and validity is based.
• If the people you intend to use a test with are very different from the people in the norming group,
then the validity and reliability evidence provided with the test becomes questionable
• Two questions to ask when reading and evaluating an empirical research article are as follows:
o “Did the researchers use appropriate measures?”
o “How much evidence did the researchers provide for measurement reliability and validity?”
o If the answers are positive, then give the article high marks for measurement. If the answers
are negative, then you should significantly downgrade your assessment of the research

Terminology used in sampling

• Sample: the set of elements selected form a population.
• Element: the basic unit selected.
• Population: the full set of elements from which the sample is selected.
• Sampling: the process of drawing a sample form a population.
• Representative sample: a sample that resembles the population.
• Equal probability of selection method (EPSEM): sampling method in which each individual element
has an equal probability of selection into the sample.
• Statistic: a numerical characteristic of sample data.
• Parameter: a numerical characteristic of a population.
• Sampling error: differences between sample values and the true population parameter.
o A key point is that some error is always present in sampling.
• Census: collection of data from everyone in the population.
• Sampling frame: a list of all the elements in a population.
• Response rate: the percentage of people selected to be in a sample who actually participate in the
research study.

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

Note: two major types of sampling used in psychological research:

1. Random sampling
2. Non-random sampling

Random sampling techniques

• When the goal is to generalize from a specific sample to a population, random sampling methods are
preferred because they produce representative samples.
• Non-random sampling methods generally produce biased samples.
• Biased sample: a non-representative sample
• It is especially important that the demographic characteristics of nonrandom samples be described in
detail in research reports so that readers can understand the exact characteristics of the research
participants
• Researchers and readers of reports can then make generalizations based on proximal similarity
• Proximal similarity: generalization to people, places, settings, and contexts that are similar to those
described in the research study.

1. Simple random sampling

• Simple random sampling: a popular and basic equal probability selection method.
2. Stratified random sampling
• Stratified random sampling: division of population elements into mutually exclusive groups and then
selection of a random sample of each group.
• Stratification variable: the variable on which the population elements are divided for the purpose of
stratified sampling.
• Stratification variables can be categorical variables or quantitative variables, and more than one
stratification variable can be used.
• How to obtain a stratified sample with just one stratification variable:
2.1 Stratify your sampling frame, and give the elements in each set identification numbers.
2.2 Draw a random sample form each of the groups
2.3 Combine the sets of randomly selected people, and you will have the final sample.
• Proportional stratified sampling: stratified sampling where the sample proportions are made to be
the same as the population proportions on the stratification variable.
• Disproportional stratified sampling: stratified sampling where the sample proportions are made to
be different from the population proportions on the stratification variable.

3. Cluster random sampling

• Cluster random sampling: sampling method where clusters are randomly selected.
• Cluster: a collective type of unit that includes multiple elements.
• One-stage cluster sampling: clusters are randomly selected and all the elements in the selected
cluster constitute the sample.
• Two-stage cluster sampling: clusters are randomly selected, and a random sample of elements is
drawn from each of the selected clusters.

4. Systematic sampling
• Systematic sampling: the sampling method where one determines the sampling interval (k),
randomly selects an element between 1 and k, and then selects every kth element.
• Sampling interval: the population size divided by the desired sample size; it’s symbolized by the letter
k.
• If you decide to draw a sample using systematic sampling, you must follow three steps.
1. Determine the sampling interval.
2. Randomly select a number between 1 and k, and include that person in your sample.
3. Also include each kth element in your sample.
• Problem in systematic sampling – periodicity
• Periodicity: problematic situation in systematic sampling that can occur if there is a cyclical pattern in
the sampling frame.

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

Non-random sampling techniques

1. Convenience sampling: use of people who are readily available, volunteer, or are easily recruited for
inclusion in a sample.
2. Quota sampling: a researcher decides on the desired sample sizes or quotas for groups identified for
inclusion in the sample, followed by convenience sampling for the groups.
3. Purposive sampling: a researcher specifies the characteristics of the population of interest and then
locates individuals who have those characteristics.
4. Snowball sampling: each sampled person is asked to identify other potential participants with the
inclusion characteristics.

Random selection and random assignment

• The purpose of random selection is to obtain a sample that represents a population.
• Random selection: selection of participants using a random sampling method.
• Random assignment: placement of participants into experimental conditions on the basis of a chance
process.
• Random assignment is not used to obtain a sample.
• Random assignment is used in experimental research to produce treatment and control groups that
are similar on all possible characteristics.
• The random assignment process is a key element in producing the strongest experimental designs
available for the study causation.
• Purpose of random selection: to obtain a representative sample.
• Purpose of random assignment: to produce two or more equivalent groups for use in an experiment.

Determining the sample size when random sampling is used

• Answers to the question: “how many people should I include in my sample?”
1. If your population is 100 people, or fewer, then include the entire population in your study
rather than drawing a sample.
2. Try to get a relatively large sample size when possible for your research study.
3. We recommend that you carefully examine other research studies in the research literature
on your topic and determine how many participants other researchers are selecting.
4. For an exact number of people to include in a sample, look at table 5.3, which shows
recommended sample sizes.
5. We strongly recommend that you use a sample size calculator
sample size calculator: a statistical program used to provide a recommended sample size
• Additional points on sample sizes:
1. You will need a larger sample size when your population is heterogeneous
2. You will need larger sample sizes when you want to break down your data into multiple
subcategories.
3. You will need larger sample sizes when you want to obtain a relatively narrow confidence
interval.
4. You will need a larger sample size when you expect weak relationships or a small effect.
5. You will need a larger sample size when you use a less-efficient technique of random
sampling.
6. Some statistical techniques require larger or smaller sample sizes.
7. You will need a larger sample size when you expect to have a low response rate.

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)

 lOMoARcPSD|17741605

Sampling in qualitative research

• Qualitative psychological research usually focuses on understanding the thinking of particular people,
groups, places, and contexts.
• A primary goal in qualitative research is to find information-rich cases.
• Sampling methods used in qualitative research:
1. Maximum variation sampling: identification and selection of a wide range of cases for data
collection and analysis.
2. Extreme case sampling: identification and selection of cases from the extreme or poles of a
dimension.
3. Homogeneous sample selection: identification and selection of a small and homogeneous
group or set of homogeneous cases for intensive study.
4. Typical-case sampling: identification and selection of what is believed to be a typical or
average case.
5. Critical-case sampling: identification and selection of particularly important cases.
6. Negative-case sampling: identification and selection of cases that you believe will probably
disconfirm your generalizations, so that you can make sure that you are not just selectively
finding cases to support your personal theory.
7. Opportunistic sampling: identification and selection of useful cases during the conduct of a
research study, as the opportunity arises.

Downloaded by Mohamad Jawad (doughman.jawad.eng@gmail.com)