FEATURE ARTICLE
PROGRAM ASSESSMENT
Q Methodology as a Tool for Program Assessment
Susan E. Ramlo
The University of Akron
Program assessment is now commonplace at most colleges and universities and
is required for accreditation of specific degree programs. Key aspects of
program assessment include program improvement, improved student learning,
and adequate student preparation for the workforce. Thus, program assessment
is a key ingredient to program health. Although surveys are often used within
program assessment in higher education, this study demonstrates the weaknesses
of this method and, instead, introduces Q methodology as a means of program
assessment especially in the area of needs assessment. In essence, Q offers an
objective way to measure subjectivity about any topic. Unlike Likert-scale
surveys, Q is a mixed method that reveals the multiple unique views as well as
consensus within the group of participants. In this study, Q was used to
determine views of a construction engineering technology program. How the
results will be used to improve the program is presented.
In higher education, there is an increasing focus on program assessment. Essentially,
program assessment is about program improvement and enhanced student learning.
Program assessment involves examining how the program impacts students and how well
program goals are met. To accomplish this, faculty are asked to demonstrate that their
programs benefit students and prepare them for the workforce (Gardiner, Corbitt, &
Adams, 2010; Martell & Calderon, 2005; McNeil, Newman, & Steinhauser, 2005). As a
result of the program assessment, programs make adjustments to better meet goals.
Program assessment leads to informed decision making, which is a key ingredient to
program health and program capabilities to meet the needs of stakeholders, including
students (Jorgensen, 2008; McNeil et al., 2005). In this way, these assessments are a
meaningful effort and not simply an attempt at appeasing the institutions’ administration
for performing program assessment, which is necessary for their institutional
accreditation and often used to determine continuation and funding of programs based on
student learning and success (Dunlap, 2008; Jorgensen, 2008). The purpose of this
program assessment of an engineering technology program was to examine students’
views about the program’s strengths and weaknesses in order to develop a plan of
continuous improvement as mandated by the program’s accrediting agency as well as the
university. In this paper, I specifically look at what the data reveals about student
perspectives about the engineering technology program and compare those views to the
program’s lead faculty member’s view. Finally, I examine the benefits of applying Q
methodology to complete this task.
Program Assessment at the University
Compliance with the accreditation standards of the Higher Learning Commission is
certainly one of the reasons for the increased focus on program assessment at universities
Mid-Western Educational Researcher • Volume 27, Issue 3
207
FEATURE ARTICLE
PROGRAM ASSESSMENT
and colleges. Program assessment typically consists of a systematic collection of
information about student learning based upon student learning outcomes at the course
and/or academic program level. Although some may focus on seeking accreditation at
the college or program level, all program assessment should use results to better inform
decisions about how to improve learning.
At my university, program assessment has been an ongoing process since about 2004.
This process is mainly at the program level and is directed by faculty with oversight
provided by our Institute for Teaching and Learning. In the recent past, I provided
program assessment for the nine engineering technology programs within my department
from 2004 until 2013. Because of the variety of programs and the differing knowledge of
program faculty about assessment, directing program assessment even at the department
level can be challenging. In addition, opinions about faculty’s programs and those of
other programs within the department are varied and sometimes contentious. This is also
true of students’ opinions about their academic programs.
A typical response would be to use a Likert-scale survey to gather information from
students. However, McKeown (2001) discusses how the use of Likert-scale surveys
results in a loss of meaning. He suggests that Q methodology offers a solution to this
problem by providing descriptive results for each perspective that emerges.
Additionally, Brown (1980) provides a description related to the conundrum of Likertscale meaning. I have adapted it for program assessment for demonstration purposes
here. Two people who respond in the same way to the same questionnaire item may
actually mean different things, or that two people responding differently may actually
mean the same thing. For example, looking at the following prompt, “This program
effectively prepares students for careers in this field." Student-1, who responds agree
strongly, may not necessarily be stronger in his agreement than Student-2 who checks
moderately agree. Their frames of reference may differ in a way such that in reality
Student-2 holds stronger opinions than Student-1. But if Student-1 says he prefers bicycle
riding (A) to baseball (B), we can be relatively more certain that A > B because of the
common frame of reference involved. This common frame of reference is, in part, what
the Q-sort, the means of collecting participant data, provides within Q methodology.
Certainly surveys are commonly used for program assessment (Gardiner et al., 2010;
Martell & Calderon, 2005), Q methodology was used as an alternative here to determine
the various views held by students about their program of study in engineering
technology and to compare those views to faculty views of these programs. The
qualitative-quantitative aspects of Q methodology represent a continuum and provide
advantages associated with using mixed methods to answer research questions that are
not present in other methods including Likert-scale surveys (Newman & Ramlo, 2010;
Ramlo & Newman, 2011).
Mid-Western Educational Researcher • Volume 27, Issue 3
208
FEATURE ARTICLE
PROGRAM ASSESSMENT
Q Methodology
In Q methodology, the statistical analyses provide Q factors. The Q factors denote
qualitative differences in perspective (Brown, 1980) and in this way the individual
statement positions help describe a kind of “world view” for each perspective. Ramlo
and Newman (2010) detailed how Q methodology can effectively be used for program
evaluation. The reasons provided by these researchers supported the choice of Q
methodology to perform a program assessment that investigated student perceptions of an
engineering technology program. Q provides descriptive profiles while also
differentiating the unique perspectives. In addition, Q reveals consensus and preserves
the meaning of the participants as they reveal their perspectives via the Q-sort. In this
way, Q best fits the purpose of the study to reveal the stakeholder perspectives of the
engineering technology program in ways that would be most meaningful for program
assessment purposes.
Q methodology was created to study subjectivity (Brown, 1980; Stephenson, 1953). This
mixed method shares many of the focuses of qualitative research while utilizing the type
of statistical analyses typically found in quantitative studies (Newman & Ramlo, 2010;
Ramlo & Newman, 2011). As Stainton-Rogers (1995) explains, compared to typical
qualitative research, Q methodology maintains the relationship among themes within the
data as it minimizes the impact of the researcher’s frame of reference. It minimizes this
impact through complex statistical analysis including correlation and factor analysis.
Despite its ability to determine the differing perspectives and consensus within a group,
Q methodology is relatively uncommon in behavioral and social science research
(Brown, 1980; Newman & Ramlo, 2010, Stephenson, 1953).
Q methodology is currently celebrating its 80th year of existence; 80 years ago
Stephenson first published an article describing Q methodology in Nature (Stephenson,
1935). As Brown (2010) stresses, Stephenson offered differentiation among some
common Q terminology. Q technique refers to the data-gathering procedure (Q sort); Q
method refers to the analytic process (factor analysis and interpretation); and
methodology denoted the conceptual and philosophical framework. Specifically, Q
methodology is a set of procedures, theory, and philosophy that focuses on the study of
subjectivity, where subjectivity is typically associated with qualitative research and
objectivity is usually associated with quantitative research (Brown, 2008; Stenner &
Stainton-Rogers, 2004).
The Q factors (views) denote qualitative differences in perspective. This interplay
between qualitative and quantitative throughout this methodology represents the reason
others have designated Q as a mixed method (Newman & Ramlo, 2010; Ramlo &
Newman, 2011; Stenner & Stainton-Rogers, 2004). Other publications have described Q
methodology in detail (Brown, 1980, 2010; McKeown & Thomas, 1988; Newman &
Ramlo, 2010; Stephenson, 1953). I will give an overview of the methodology here
within the context of this particular study.
Mid-Western Educational Researcher • Volume 27, Issue 3
209
FEATURE ARTICLE
PROGRAM ASSESSMENT
Developing Concourse for the Program Assessment
Any Q methodology study commences with the development of the concourse which is a
collection of items, typically statements, about the topic that have been collected by the
researcher (McKeown & Thomas, 1988). For this program assessment, the concourse
was developed from a variety of sources. First, each engineering technology program
within the department has a strategic plan and program learning outcomes. Next, most of
the programs are accredited by the Technology Accreditation Commission (TAC) of
ABET and must meet certain criteria. Data for accreditation includes self-studies, alumni
surveys, and a collection of student learning assessments. These materials were used to
provide statements for the concourse. In addition, I performed informal interviews with
faculty and students to collect views about programs that may not be part of strategic
plans or self-studies. The concourse items were sent out to faculty for feedback and
several other statements were added to the concourse. This feedback was also used to
help select the 48 statements for the Q-sample which is a subset of the concourse that
represents the communications on the topic. Faculty and students sorted the Q-sample
based upon their view of their engineering technology program.
Figure 1. Sorting grid for this study
2
3
4
4
5
Most
unlike
my
view
-5
6
5
4
4
3
Most
like
my
view
neutral
-4
-3
-2
-1
0
2
1
2
3
4
5
Sorting of the Q sample
Participants sorted the 48-item Q-sample into a grid provided by the researcher and
displayed in Figure 1. The condition of instruction was for sorters to arrange the items
based upon their views of their engineering technology program major. Faculty provided
sorts and asked students within capstone courses to provide sorts. In most cases,
classroom time was given for the student sorting. Only the results for the Construction
Engineering Technology program, which consists of an Associate Degree and a Bachelor
Degree, are provided within this manuscript.
Mid-Western Educational Researcher • Volume 27, Issue 3
210
FEATURE ARTICLE
PROGRAM ASSESSMENT
Analyses
Thirty-five participants from the Construction Engineering Technology (CET) program
sorted the 48 statements from the Q-sample. These statements are provided in Table 1
along with results of the analyses which will be described in a subsequent section of this
paper. The study’s participants consisted of one faculty member, 19 bachelor degree
students, and 15 students in the associate degree CET program. The initial analysis of the
Q sorts is essentially a correlation using factor analysis. A factor matrix is created and
participants are flagged when associated with a specific factor. Participants are identified
on one factor based on the factor loadings. Only those participant Q-sorts identified with
a factor are used to describe that factor in subsequent analyses.
Results
Three distinct views (factors) emerged from the factor analysis. Factor 3 includes the
lead-faculty person in addition to five Bachelor (BS) degree students and six Associate
(AAS) degree students. Two of these BS students have negative Factor 3 scores. This
means that these two students are negatively correlated with this view/factor.
Factor 1 consists of 11 students – seven at the BS level and four at the AAS level. Six
student views are represented by Factor 2 with two BS students and four AAS students.
These students all have positive factor scores on their representative factor. Note that all
of the students except one are male; the female student is an AAS student represented by
the Factor 3 view along with the faculty member who is also a female.
Table 1
Those statements that distinguish Factor 1 from the other factors
No.
Statement
Factor 1
Grid
Position
Factor 2
Grid
Position
Factor 3
Grid
Position
1^
This program provides students a quality
education.
3
2
4
2
This program effectively prepares students
for careers in this field.
0*
3
2
3
This program teaches students how to be
problem solvers within the context of this
field.
-1
1*
-2
4
Instructors within this program do all they
can to enhance students' learning.
2*
0*
4*
(continued)
Mid-Western Educational Researcher • Volume 27, Issue 3
211
FEATURE ARTICLE
PROGRAM ASSESSMENT
Table 1 (continued)
Those statements that distinguish Factor 1 from the other factors
No.
Statement
Factor 1
Grid
Position
Factor 2
Grid
Position
Factor 3
Grid
Position
5
Most instructors outside of this specific
program provide an environment that fosters
student learning.
-3
-3
1*
6^
This program provides students with the
skills to continue learning after they have
graduated.
2
3
2
-3*
3*
0*
-5*
-1
0
1
0
0
0
0
2*
7
This program provides opportunities for
students to apply what they are learning in
the classroom in real-world settings.
8
Equipment used within the program is
appropriate and state of the art.
9^
Faculty within this program cultivate within
students a strong ethical commitment to the
field.
10
This program effectively prepared students to
perform mathematical analyses /
calculations.
11
This program focused on conceptual
understanding of topics related to the field.
1
0*
2
12
This program and its faculty stressed the
importance of professionalism inside and
outside the workplace.
0
-1
1
13
Faculty within this program were well
qualified to teach because they were
knowledgeable about the subjects.
3*
0*
5*
14
Faculty within this program were good
teachers and helped students learn.
1
1
5*
15^
Having this program accredited by a national
organization is important for students and
their careers.
5
4
3
16
Faculty were interested in students' learning.
1
1
3*
(continued)
Mid-Western Educational Researcher • Volume 27, Issue 3
212
FEATURE ARTICLE
PROGRAM ASSESSMENT
Table 1 (continued)
Those statements that distinguish Factor 1 from the other factors
Factor 1
Grid
No.
Statement
Position
Factor 2
Grid
Position
Factor 3
Grid
Position
17^
Topics addressed within this program are
appropriate and current/up-to-date.
2
0
1
18
Students learn how to function in a diverse
workplace through their course work at the
University.
0
-3*
-1
-2
-1*
-2
19
Participation of students in student groups
within the program is important for student
success.
20
This program needs better facilities
(classrooms, laboratories, etc.).
5*
-2*
-4*
21
This program would be more appealing to
prospective students if it was not in Summit
College.
4*
-1*
-5*
22^
I would recommend this program to
prospective students.
4
5
3
23
The university supports this program at an
appropriate level.
-4
-3
-1
24
The program's course schedule works well
for students.
-3*
-5*
0*
25
Students in this program are good at
interpreting information presented in a
variety of visual forms (drawings, graphs,
etc.) in a variety of contexts.
-1
2*
-1
26
Students in this program are effective at
presenting data / information.
-1
2*
-2
27
Students in this program have learned to
work effectively in teams both in academia
and in the workplace.
0*
2*
1*
28
Students in this program can work
effectively as individuals in academia and in
the workplace.
1
4*
0
(continued)
Mid-Western Educational Researcher • Volume 27, Issue 3
213
FEATURE ARTICLE
PROGRAM ASSESSMENT
Table 1 (continued)
Those statements that distinguish Factor 1 from the other factors
Factor 1
Grid
No.
Statement
Position
Faculty possess the type of technical
expertise needed to teach courses in this
29
1
program.
30
Students in this program can perform the
types of experiments/tests/data
collection/data analysis required within this
field.
Factor 2
Grid
Position
Factor 3
Grid
Position
2
4*
-2*
0*
2*
31
Students who complete this program are
capable of managing projects related to the
field.
0*
4*
-3*
32
Having this program and its courses offered
on the main campus of the University is
important to students.
3*
-2*
0*
-1
-1
-4*
3
3
3
0*
-5*
-3*
-2
-1
-2
-1
-2
-1
33
34^
35
36^
37^
There is a lot of redundancy within this
program (e.g. same or similar topics are
repeated throughout the courses within the
program).
Program faculty are up to date in their
knowledge of their field and bring that
knowledge to the classroom.
Program courses need to be updated so that
they are more in line with current industry
practice as you see it.
This program offers sufficient opportunities
for students to come into contact with other
professionals in the field (who do not teach
at this university).
Student groups related to this program offer
sufficient opportunities for learning more
about this field.
(continued)
Mid-Western Educational Researcher • Volume 27, Issue 3
214
FEATURE ARTICLE
PROGRAM ASSESSMENT
Table 1 (continued)
Those statements that distinguish Factor 1 from the other factors
Factor 1
Grid
No.
Statement
Position
Factor 2
Grid
Position
Factor 3
Grid
Position
38
There is not enough mathematical rigor
within this program.
-5*
-4*
-1*
39
Students need to perform more written and
oral presentations within this program to be
prepared for their professions.
-4
-4
-2*
40^
Program faculty are well qualified to teach
within this program.
2
1
1
41
Students within this program learn how to
write clear and effective engineering
technology-related reports.
-2*
1
0
42^
Students within this program learn how to
effectively present information orally
(speech).
-4
-2
-3
0
0
-1
2*
-2*
1*
43^
44
Students within this program can effectively
use software to address technical problems
and analyze data.
Having this program recognized locally as a
quality program is important for students &
alumni.
45
I wish this program also had a graduate
program associated with it.
4*
1*
-5*
46
This program makes students highly
employable and prepared for the workforce.
-1*
5*
0*
47^
The resources available at the university
(sports programs, library, etc.) are important
to this program and its students.
-3
-4
-3
It would be better if this program was in a
-2
-3
-4
college focused on just technology or
engineering.
Note: *Denotes distinguishing statement for that factor; ^designates consensus statement
48^
Table 1 summarizes all of the findings produced from the analyses. Separate tables
include characteristic-sorts for each factor, distinguishing statements, and consensus
Mid-Western Educational Researcher • Volume 27, Issue 3
215
FEATURE ARTICLE
PROGRAM ASSESSMENT
statements. Interpretation of each factor is based upon those statements that those
represented by these factors felt the most strongly about (as indicated by grid positions
provided for each factor as displayed – focusing on statements at the +4, +5, -4, and -5
positions as displayed in Table 1). Distinguishing statements, which differentiate one
factor from the others, are also important for interpretation of each of the factors.
Factor 1/View 1: A good program that needs improvements in certain areas
Overall, Factor 1/View 1 has a positive attitude about the Construction Engineering
Technology program but believes that the program needs improved facilities, more
support from the university, the addition of a graduate degree, and a position within a
different college at the university.
Those represented by Factor 1/View 1 see the program’s accreditation as important for
students and their careers. Although they do not believe that students need to perform
more written and oral presentations within their program, they do not believe that
construction students learn how to effectively present information orally. Those
represented by this view believe there is enough mathematical rigor within this program.
This description is further supported by those statements that distinguish this factor from
the other two, as indicated by asterisks in Table 1 within the Factor 1 grid-position
column. However, these distinguishing statements reveal that those represented by View
1 also hold a neutral position about how well the program prepares students for their
careers (statement 2) including project management (statement 31), and employability
(46). This seems to coincide with statement 7 (at -3) that indicates this view does not
believe the program provides enough application of construction in real-world
applications. Thus, the Factor 1/View 1 was named “A good program that needs
improvements in certain areas."
Factor 2/View 2: Practical Students - program that prepares students for the
construction industry
In Table 1, the column labeled “Factor 2 Grid Position” provides information about
statement location and distinguishing statements for the Factor 2 view. Factor 2/View 2
represents six students in the Construction Engineering Technology Program. As Brown
(1980) details, having six Q sorts on a factor is sufficient to provide a stable factor and
ensuing description. Like Factor 1/View 1, those represented by this view possess a
positive view of the Construction program (Statement 22 at +5) and of program
accreditation (Statement 15 at +4). However, this view appears focused on the program’s
preparation of students to work in the construction industry. In other words, they appear
more career-centered. Factor 2 students do not believe the construction courses do not
prepare them to work in a diverse work place (Statement 18 at -3, a distinguishing
statement). These students believe the construction program prepares them to manage
projects (Statement 31 at +4, distinguishing) and work effectively as individuals
(Statement 28 at +4, distinguishing) in academia and in the workplace. They believe the
CET program makes students highly employable and prepared for the workforce.
Typical university resources like sports and libraries are unimportant to those represented
Mid-Western Educational Researcher • Volume 27, Issue 3
216
FEATURE ARTICLE
PROGRAM ASSESSMENT
by this view (statement 47 at -4). Those represented by the Factor 2 view expressed
problems with the scheduling of courses (Statement 24 at +5) but satisfaction with the
level of math in the program as well as oral and written presentations. Unlike the Factor
1/View 1 students, students representing this view believe the program provides
opportunities for students to apply what they are learning in the classroom in real-world
settings (Statement 7, +3 and distinguishing). Those represented by Factor 2/View 2
believe the construction program is in line with current industry practices (Statement 35
at -5). Factor 2/View 2 was named “Practical students - program prepares students for
the construction industry.”
Factor 3/View 3: Program faculty make the program good
The Factor 3/View 3 grid position column in Table 1 indicates the location of the
statements for the representative sort for this factor/view. Like the other two views,
Factor 3/View 3 sorters agree that the construction program provides a quality education
(Statement 1 at +4). The remaining four of the ‘most like’ statements (+5 and +4 grid
positions) start with the word “Faculty” or “Instructors” so it is easily seen that this view
is most focused on the quality of instruction within the construction engineering
technology program (Statements 4, 13, 14, and 29). Statement 16 is at +3 and
distinguishing (Faculty were interested in students' learning). Those represented by the
Factor 3 view disagree that the program needs better facilities (Statement 20 at -4,
distinguishing) and that moving the program to a different college would make it more
appealing to prospective students (Statement 21 at -5, distinguishing; Statement 48 at -4).
Factor 3/View 3 representatives also disagree that they are interested in creating a
graduate program associated with the construction engineering technology degree
(Statement 45 at -5).
It is important to note that this was the factor that represents the lead- faculty person from
the CET program (positive factor loading). It is also important to remind readers that this
is also the factor which is bipolar in that there were both positive (10) and negative (2)
loaders. Thus the negative loaders have a negative view of the program faculty, believe a
graduate degree is desirable, and believe the facilities need to be upgraded, for instance.
This factor was named “Program faculty make the program good.”
Consensus among the views
Along with representative sorts and distinguishing statements, the Q analyses produce a
table of consensus. These are the statements that do not discriminate among the pairs of
factors. In other words, these are the statements that the three differing views agreed
upon at various levels, based upon grid position which is also provided in the table.
Table 1 contains the consensus statements for this study and they are indicated with a ‘^’
sign. Here we see general agreement that the program provides students the skills to
continue to learn after graduation (statement 6) and the importance of maintaining the
program’s accreditation (statement 15). All three factors have a grid position of +3 for
statement 34 (Program faculty are up to date in their knowledge of their field and bring
that knowledge to the classroom). However, the three views are neutral about statement
Mid-Western Educational Researcher • Volume 27, Issue 3
217
FEATURE ARTICLE
PROGRAM ASSESSMENT
9 (Faculty within this program cultivate within students a strong ethical commitment to
the field) and statement 37 (Student groups related to this program offer sufficient
opportunities for learning more about this field). Students are also in agreement across
all three perspectives that they do not learn how to effectively make oral presentations
(statement 42). They agree that the program does not provide sufficient opportunities for
them to network with professionals in the field (statement 36).
Limitations of the Study
Typically, generalizability is a desirable goal of social science research. However, Q is
not generalizable in the typical sense of that term. Thomas and Baas (1993) distinguish
two types of generalizability in social science research by focusing on two types of
generalizability: statistical inference and substantive inference. The more typical
generalizability would be statistical inference, where the purpose is generalizing to a
larger audience from a large, random sample of participants. Q methodology, however,
uses substantive inference, where the focus is a more qualitative one about the about
phenomenon (Thomas & Baas, 1993). In Q methodology, Q factors represent
generalizations about how persons of a certain perspective think about the topic under
investigation (Brown, 1980; Thomas & Baas, 1993). In other words, generalizations in Q
relate to general principles such as the relations of and between factors (Brown, 1980).
All sorters were asked to comment on their “most like” and “most unlike” statement
placements as well as their decisions related to the sort. I used these questions in prior
studies and gained additional insight into the sorters’ views. However, in this study the
students’ comments were minimal and sometimes missing. Not much insight, therefore,
was gained by reviewing students’ written comments and that is why they are not
included in the descriptions of the factors views. Because participation was anonymous,
it was not possible to follow up with sorters with interviews or other means.
Discussion
Classifying individuals into different perspectives (profiles) is helpful in various research
situations especially in applications where different groups may be affected differently by
programs (McNeil et al., 2005). In program assessment it is frequently important to
address the various stakeholder groups differently to ascertain their needs; more
successfully addressing stakeholder needs improves the effectiveness of the program and
makes the recommendations more likely to be implemented (McNeil et al., 2005; Ramlo
& Newman, 2010). Within this study, I have demonstrated how Q methodology can be
used to describe these different perspectives in ways that can provide insight for program
assessment much like that discussed by Ramlo and Newman (2010). The development of
such profiles is necessary for effective program assessment and allows researchers to
better inform stakeholders about group differences and to make improvements to address
different groups’ needs. .
In this study, three unique views emerged from analyzing the Q sorts of 35 participants.
The first view, Factor 1/View 1, agreed that the Construction Engineering Technology
Mid-Western Educational Researcher • Volume 27, Issue 3
218
FEATURE ARTICLE
PROGRAM ASSESSMENT
(CET) program was a good one but they also wanted to see changes. Many of these
changes were administrative such as the location of the programs within the university
structure, funding, facilities, and the addition of a graduate degree. As far as instruction,
this view believes that more real-world applications are necessary to better prepare
students and their learning. Factor 1/View 1 consisted of 11 students – seven at the
bachelor level and four at the associate level.
Factor 2/View 2 was more focused on the ability of the program to prepare students for
the workforce and suggested that the CET program is doing a good job at this. Six
students were represented by Factor 2. They disagreed with Factor 1/View 1 and
indicated that faculty include sufficient real-world applications in their classrooms. They
did agree with Factor 1/View 1 that the scheduling of classes is a problem. Those
represented by Factor 2/View 2 had a more neutral view of faculty than the Factor
3/View 3 which was very faculty focused.
Factor 3/View 3 included the one female student-participant and the one faculty member
participant. This factor represents 12 sorters total including five bachelor degree students
and six associate degree students along with the faculty member. Of the five bachelor
degree students, two have negative loadings on the factor. This means these two sorters
have an opposing view relative to the positive sorters. In other words, whereas this factor
is focused positively on the program faculty as key to the program’s success, the two
negative loaders would view program faculty negatively.
Overall, it is helpful to see how different stakeholders view the same program. Consensus
in this study is also helpful because it helps us see the agreement among the sorters that
CET students need to become better at oral presentations. Based on these findings, the
researcher suggests that the next round of program assessment focus on examining the
number and quality of presentations made within the CET program classes. Because
written and oral communications are key program learning outcomes for the program
based upon the accreditation agency, this appears to be an important but weak area within
the program.
Other insight revealed that, although there is no consensus among all three factors,
investigating the addition of a graduate program, improvements to labs, increased
funding, etc. are also areas of potential future focus. These specific areas call for the
college and university administration involvement because they require an investment of
resources.
Conclusion
The results from this study resulted in programmatic changes but also continued requests
for improved support from the larger university. Those represented by Factor 1/View 1
believe that the Construction Engineering Technology program is good but needs
improved facilities, more support from the university, the addition of a graduate degree,
and a position within a different college at the university. Although course-fee money
helped to update some of the program’s laboratory facilities, larger university financial
support is still wanting. New university leadership has encouraged the restarting of
Mid-Western Educational Researcher • Volume 27, Issue 3
219
FEATURE ARTICLE
PROGRAM ASSESSMENT
conversations regarding a master degree program that would serve the various bachelor
degree programs in engineering technology and applied sciences. The program remains
in the same college but that college in the midst of change including a new college name
and new mission that is more in line with programs such as Construction Engineering
Technology.
Factor 2/View 2 students believe that the program can be improved with a greater focus
on the preparation of students to work in the construction industry. The Construction
Engineering Technology program has since increased service-learning opportunities as
well as other work-experiences to address this view’s belief on how to improve the
program. Additionally, a department-wide Software Applications course has been
replaced in the curriculum with one that is specifically designed for construction students
with associated applications throughout.
Students and the faculty-member represented by Factor 3/View 3 believed that the
construction faculty were key to student learning. Their belief that students did not
experience sufficient project management preparation is addressed by the inclusion of
more construction experiences that include service-learning, as previously mentioned.
Overall, this study also provided evidence that the program is focused on continuous
improvements for a recent accreditation visit as well as program self-study. Upon
completion of this study, the results were shared as part of the university-wide program
assessment process and satisfied that initiative’s focus on assessment and evidence of
continuous improvement.
In addition, results of this study indicate that leaders of student groups in CET may want
to investigate how they can improve students’ learning and application of construction
knowledge to real-world tasks or for networking within the local, regional, or national job
market. Currently some of these student groups participate in competitions that involve
program specific applications such as estimating and it may be possible to expand this
type of involvement, perhaps expanding into other competitions. Construction faculty
need to be cognizant of students’ desire for real-world tasks within the classroom as well
as career networking. Finally, while the university decreases the overall number of
tenure-track faculty positions across all programs, the results of this study reveal
students’ belief that maintaining the quality of program faculty, tenure-track and
adjuncts, is necessary for upholding the quality of the CET program instruction.
Future Research
In the future, additional stakeholders including alumni, employers of graduates, and the
program industrial advisory committee, should also participate in the program
assessment. Broader participation will also bring with it a need for a revised Q-sample
that better matches the purpose of such a study. The ability to perform Q sorting offsite
including the possibility of online Q sorting will need to be investigated. In addition,
considering the brief comments written by student participants in this study, future
program assessment should include interviews of the all sorters in order to further clarify
their perspectives.
Mid-Western Educational Researcher • Volume 27, Issue 3
220
FEATURE ARTICLE
PROGRAM ASSESSMENT
Author Notes
Susan E. Ramlo is a Professor of General Technology-Physics and Professor of Physics
at The University of Akron.
Correspondence concerning this article should be addressed to Susan E. Ramlo at
sramlo@uakron.edu
Mid-Western Educational Researcher • Volume 27, Issue 3
221
FEATURE ARTICLE
PROGRAM ASSESSMENT
References
Brown, S. R. (1980). Political subjectivity: Applications of Q methodology in political
science. New Haven, CT: Yale University Press.
Brown, S. R. (2008). Q methodology. In L. M. Given (Ed.), The Sage encyclopedia of
qualitative research methods (pp. 700-704). Thousand Oaks, CA: Sage Publications
Brown, S. R. (2010). Q methodology. In N. J. Salkind (Ed.), Encyclopedia of research
design (pp. 1149-1155). Thousand Oaks, CA: Sage.
Dunlap, L. A. (2008). An effective approach to generating questions for guiding program
assessment and reform. Academic Leader, 24(10), 1-6.
Gardiner, L. R., Corbitt, G., & Adams, S. J. (2010). Program assessment: Getting to a
practical how-to model. Journal of Education for Business, 85(3), 139-144.
Jorgensen, M. (2008). Have you been flossing? An analogy for educational program
assessment. Assessment Update, 20(5), 9-10.
Martell, K. D., & Calderon, T. G. (2005). Assessment of student learning in business
schools: Best practices each step of the way. Tallahassee, FL: Association for
Institutional Research; AACSB International.
McKeown, B. (2001). Loss of meaning in Likert scaling: A note on the Q methodological
alternative. Operant Subjectivity, 24, 201-206.
McKeown, B., & Thomas, D. (1988). Q methodology. Newbury Park, CA: Sage
Publications.
McNeil, K. A., Newman, I., & Steinhauser, J. (2005). How to be involved in program
evaluation: What every administrator needs to know. Lanham, Maryland: Scarecrow
Education.
Newman, I., & Ramlo, S. (2010). Using Q methodology and Q factor analysis to facilitate
mixed methods research. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed
methods in social & behavioral research (2nd ed., pp. 505-530). Thousand Oaks,
CA: Sage Publications.
Ramlo, S., & Newman, I. (2010). Classifying individuals using Q methodology and Q
factor analysis: Applications of two mixed methodologies for program evaluation.
Journal of Research in Education, 20(2), pp 20-31. Retrieved from
http://www.eeraonline.org/journal/files/v20/JRE_v20n2_Article_3_Ramlo_and_Ne
wman.pdf
Mid-Western Educational Researcher • Volume 27, Issue 3
222
FEATURE ARTICLE
PROGRAM ASSESSMENT
Ramlo, S.,& Newman, I. (2011). Q methodology and its position in the mixed methods
continuum. Operant Subjectivity: The International Journal for Q Methodology, 34
(3), pp. 173-192. doi:10.15133/j.os.2010.009
Stainton-Rogers, R. (1995). Q methodology. In J. A. Smith, R. Harré, & L. van
Langenhove (Eds.), Rethinking methods in psychology (pp. 178-192). London;
Thousand Oaks, CA.: Sage Publications.
Stenner, P., & Stainton-Rogers, R. (2004). Q methodology and qualiquantology: The
example of discriminating between emotions. In Z. Todd, B. Nerlich, S. McKeown
& D. D. Clarke (Eds.), Mixing methods in psychology. Hove; New York: Psychology
Press.
Stephenson, W. (1935). Technique of factor analysis. Nature, 136, 297.
Stephenson, W. (1953). The study of behavior: Q-technique and its methodology.
Chicago: University of Chicago Press.
Thomas, D.D., & Baas, L.R. (1993). The issue of generalization in Q methodology:
“Reliable Schematics” revisited. Operant Subjectivity, 16, 18-36.
Mid-Western Educational Researcher • Volume 27, Issue 3
223