Focus groups are a popular qualitative method tool and are particularly useful in program evaluat... more Focus groups are a popular qualitative method tool and are particularly useful in program evaluation. Participants, who are guided by a trained facilitator, engage in and co-construct a dialogue surrounding a topic of interest. This interaction produces a richness of data that cannot be captured in individual interviews, and that can become a powerful learning tool for all parties involved. This demonstration will describe how to design and facilitate engaging focus groups. The presenter will share two unique focus group formats that were intentionally designed and carried out in a longitudinal evaluation of a general education program at a large research university. The presenter will highlight the benefits of using focus groups in program evaluation and practical strategies for selecting an appropriate focus group format and moderating the discussion. In addition, a brief description of the analysis and results will be shared.
Too often we are faced with the challenge of collecting survey responses in order to hit our targ... more Too often we are faced with the challenge of collecting survey responses in order to hit our target response rate. This demonstration will walk through practical tips and techniques that can be used to increase survey responses, particularly for telephone surveys and online surveys. The presenter will guide attendees through a step-by-step process of how to effectively plan, implement, and track respondents in survey research. These steps can be considered the “moves” one makes, as an evaluator or researcher, in hopes of generating many authentic and accurate responses. The presenter will use examples from two different education evaluation studies that aimed to reach between 250 to over 1,000 clients.
The focus of this study is on responses from an interactive focus group session conducted in 2014... more The focus of this study is on responses from an interactive focus group session conducted in 2014 with 62 undergraduate students at the University of Hawaiʻi at Mānoa (UHM). These students were part of a longitudinal study running from 2010-2015. A sample of 251 first-time undergraduate students was recruited from the fall 2010 incoming class through a combination of convenience and random sampling. Each year, students submitted six online surveys with open- and closed-ended questions, participated in an annual focus group, and submitted coursework each semester. The results for this paper come from the 4th year focus group, which was guided by three questions: (1) what helps learning in general education? (2) what factors and experiences are most important?, and (3) how, if at all, factors and experiences differ across the general education components?
Q methodology captures individuals’ points-of-view, reveals their understandings of issue(s), and... more Q methodology captures individuals’ points-of-view, reveals their understandings of issue(s), and produces a set of consensus points, compromise points, and non-consensual and non-confrontational points. Q methodology is a mixed-method tool that combines qualitative data collection methods and quantitative data analysis. This paper will describe Q methodology as an ideal tool in program evaluation because it includes the complexity of individual perspectives. The presenters will describe how they carried out Q methodology as part of an evaluation of a general education program at a large research university. We will highlight how the results provided actionable data for program improvement. In addition, we found that Q methodology can reveal whether points-of-views differ by program component and thus Q methodology can be used to investigate whether stakeholders’ perspectives change when asked about different program components or contexts using the same set of “Q statements.”
The graduate student experience is unique to each institution with one significant factor being t... more The graduate student experience is unique to each institution with one significant factor being the community of its programs. This program will begin with an introduction to the student organization to be discussed throughout the presentation, the Higher Education Student Association (HESA) at the University of Hawai‘i at Manoa. This will include its mission, vision and purpose as well as its composition and logistics to the organization. We will then dive into a historical overview of HESA and why it first came about and describe insights and challenges that were initially faced. A present day look into HESA will then be covered that will discuss today’s goals and priorities of HESA, and how it has changed over time and why. The presentation will end looking beyond where we are today, including going over hopes for the future of HESA, where we see it in the next five years, as well as implications and directions for other colleges/departments to model after in creating and sustaining their own student organizations.
This demonstration will walk attendees through the process of planning and conducting phone surve... more This demonstration will walk attendees through the process of planning and conducting phone survey research, which is a widely-used method in evaluation. The presenter will illustrate the process by describing the phone survey component of an evaluation study on college student retention that included cold-calling over 1,000 clients, asking personal questions, and recording the information in a useable form.
The presenter will begin by briefly describing the purpose of using phone surveys (i.e., when to use phone surveys versus other data collection methods) and its advantages and disadvantages. Attendees will then be guided through a step-by-step process of how to (1) develop an effective phone survey (e.g., survey software, questions, length, flexibility, etc.), (2) establish a script or protocol to use when speaking to respondents, (3) create a tracking sheet to track respondents, and (4) appropriately communicate with respondents while conducting the survey (e.g., voice, tone, and persona, being adaptable, probing and clarification, etc.)
Throughout the demonstration, practical tips and advice that address ongoing challenges of phone survey research such as response rates, call scheduling, and reliability and validity will be shared. The presenter will also suggest techniques for addressing new and emerging challenges such as issues with caller ID, voicemail, privacy and confidentiality, and respondent burden.
Institutions that understand what helps and hinders student learning will know how to improve stu... more Institutions that understand what helps and hinders student learning will know how to improve student proficiency in core academic areas that are important for college and workforce success. Learning is a complex, developmental process (Beyer et al., 2007). Specific to the university setting, it is evident from post-1990 research on the effects of college that learning is holistic; just as a single learning experience may affect multiple facets of students’ lives, multiple experiences may influence a single learning outcome (Pascarella and Terenzini, 2005). This sparked efforts to employ innovative pedagogical approaches and encourage principles of learning that are domain-independent, experience-independent, and culturally relevant (Ambrose et al., 2010). Faculty and administrators must continue to examine a variety of (new) learning tools for all students to improve or evolve undergraduate teaching methods, curriculum, and experiences. The purpose of this poster is to document and compare student perceptions over time in regards to factors and experiences that help and hinder student learning at one university campus.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
Beyer, C. H., Gillmore, G. M., & Fisher, A. T. (2007). Inside the Undergraduate Experience: The University of Washington's Study of Undergraduate Learning (JB-Anker Series).
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). K. A. Feldman (Ed.). San Francisco: Jossey-Bass.
The University of Hawai‘i at Mānoa’s strategic plan includes being a “Hawaiian Place of Learning.... more The University of Hawai‘i at Mānoa’s strategic plan includes being a “Hawaiian Place of Learning.” In this presentation we describe our longitudinal study (2010-2014) that captured student perceptions related to Hawaiian Place of Learning (HPL). We present results such as students’ HPL definitions, HPL’s importance to them, extent to which they view the university as a HPL, and their self-reports of amount learned about Native Hawaiian culture and where that learning took place. In addition, we discuss the strengths and challenges of a longitudinal research design and next steps regarding use of results.
This study explored college student perceptions of racial and ethnic stereotype-based humor in Ha... more This study explored college student perceptions of racial and ethnic stereotype-based humor in Hawai’i where humor is salient to the local culture. It has been suggested elsewhere that perceptions may be understood through eliciting a cultural domain or mental category including a set of items that are perceived to belong in the same category. For the present study, 72 students participated in interviews or focus groups to explore characterizations of racial and ethnic stereotype-based humor among university students in Hawai’i. The interviews consisted of generating free lists, rank orders and pile sorts, and were analyzed using ANTHROPAC software to convert textual data into matrices and cluster analyses to interpret the cultural domain. The findings indicated that students’ understanding of local stereotypes and humor aligned with the conceptualization of a social stratification in Hawai’i, because they acknowledged more derogatory stereotypes for certain racial and ethnic populations. As colleges continue to work towards diverse learning environments, humor is likely to be one of the more complex areas to address. Future research may involve a content analysis that examines campus fliers, events and media to understand better the role of the institution in developing and perpetuating existing stereotypes.
Focus groups are a popular qualitative method tool and are particularly useful in program evaluat... more Focus groups are a popular qualitative method tool and are particularly useful in program evaluation. Participants, who are guided by a trained facilitator, engage in and co-construct a dialogue surrounding a topic of interest. This interaction produces a richness of data that cannot be captured in individual interviews, and that can become a powerful learning tool for all parties involved. This demonstration will describe how to design and facilitate engaging focus groups. The presenter will share two unique focus group formats that were intentionally designed and carried out in a longitudinal evaluation of a general education program at a large research university. The presenter will highlight the benefits of using focus groups in program evaluation and practical strategies for selecting an appropriate focus group format and moderating the discussion. In addition, a brief description of the analysis and results will be shared.
Too often we are faced with the challenge of collecting survey responses in order to hit our targ... more Too often we are faced with the challenge of collecting survey responses in order to hit our target response rate. This demonstration will walk through practical tips and techniques that can be used to increase survey responses, particularly for telephone surveys and online surveys. The presenter will guide attendees through a step-by-step process of how to effectively plan, implement, and track respondents in survey research. These steps can be considered the “moves” one makes, as an evaluator or researcher, in hopes of generating many authentic and accurate responses. The presenter will use examples from two different education evaluation studies that aimed to reach between 250 to over 1,000 clients.
The focus of this study is on responses from an interactive focus group session conducted in 2014... more The focus of this study is on responses from an interactive focus group session conducted in 2014 with 62 undergraduate students at the University of Hawaiʻi at Mānoa (UHM). These students were part of a longitudinal study running from 2010-2015. A sample of 251 first-time undergraduate students was recruited from the fall 2010 incoming class through a combination of convenience and random sampling. Each year, students submitted six online surveys with open- and closed-ended questions, participated in an annual focus group, and submitted coursework each semester. The results for this paper come from the 4th year focus group, which was guided by three questions: (1) what helps learning in general education? (2) what factors and experiences are most important?, and (3) how, if at all, factors and experiences differ across the general education components?
Q methodology captures individuals’ points-of-view, reveals their understandings of issue(s), and... more Q methodology captures individuals’ points-of-view, reveals their understandings of issue(s), and produces a set of consensus points, compromise points, and non-consensual and non-confrontational points. Q methodology is a mixed-method tool that combines qualitative data collection methods and quantitative data analysis. This paper will describe Q methodology as an ideal tool in program evaluation because it includes the complexity of individual perspectives. The presenters will describe how they carried out Q methodology as part of an evaluation of a general education program at a large research university. We will highlight how the results provided actionable data for program improvement. In addition, we found that Q methodology can reveal whether points-of-views differ by program component and thus Q methodology can be used to investigate whether stakeholders’ perspectives change when asked about different program components or contexts using the same set of “Q statements.”
The graduate student experience is unique to each institution with one significant factor being t... more The graduate student experience is unique to each institution with one significant factor being the community of its programs. This program will begin with an introduction to the student organization to be discussed throughout the presentation, the Higher Education Student Association (HESA) at the University of Hawai‘i at Manoa. This will include its mission, vision and purpose as well as its composition and logistics to the organization. We will then dive into a historical overview of HESA and why it first came about and describe insights and challenges that were initially faced. A present day look into HESA will then be covered that will discuss today’s goals and priorities of HESA, and how it has changed over time and why. The presentation will end looking beyond where we are today, including going over hopes for the future of HESA, where we see it in the next five years, as well as implications and directions for other colleges/departments to model after in creating and sustaining their own student organizations.
This demonstration will walk attendees through the process of planning and conducting phone surve... more This demonstration will walk attendees through the process of planning and conducting phone survey research, which is a widely-used method in evaluation. The presenter will illustrate the process by describing the phone survey component of an evaluation study on college student retention that included cold-calling over 1,000 clients, asking personal questions, and recording the information in a useable form.
The presenter will begin by briefly describing the purpose of using phone surveys (i.e., when to use phone surveys versus other data collection methods) and its advantages and disadvantages. Attendees will then be guided through a step-by-step process of how to (1) develop an effective phone survey (e.g., survey software, questions, length, flexibility, etc.), (2) establish a script or protocol to use when speaking to respondents, (3) create a tracking sheet to track respondents, and (4) appropriately communicate with respondents while conducting the survey (e.g., voice, tone, and persona, being adaptable, probing and clarification, etc.)
Throughout the demonstration, practical tips and advice that address ongoing challenges of phone survey research such as response rates, call scheduling, and reliability and validity will be shared. The presenter will also suggest techniques for addressing new and emerging challenges such as issues with caller ID, voicemail, privacy and confidentiality, and respondent burden.
Institutions that understand what helps and hinders student learning will know how to improve stu... more Institutions that understand what helps and hinders student learning will know how to improve student proficiency in core academic areas that are important for college and workforce success. Learning is a complex, developmental process (Beyer et al., 2007). Specific to the university setting, it is evident from post-1990 research on the effects of college that learning is holistic; just as a single learning experience may affect multiple facets of students’ lives, multiple experiences may influence a single learning outcome (Pascarella and Terenzini, 2005). This sparked efforts to employ innovative pedagogical approaches and encourage principles of learning that are domain-independent, experience-independent, and culturally relevant (Ambrose et al., 2010). Faculty and administrators must continue to examine a variety of (new) learning tools for all students to improve or evolve undergraduate teaching methods, curriculum, and experiences. The purpose of this poster is to document and compare student perceptions over time in regards to factors and experiences that help and hinder student learning at one university campus.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
Beyer, C. H., Gillmore, G. M., & Fisher, A. T. (2007). Inside the Undergraduate Experience: The University of Washington's Study of Undergraduate Learning (JB-Anker Series).
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). K. A. Feldman (Ed.). San Francisco: Jossey-Bass.
The University of Hawai‘i at Mānoa’s strategic plan includes being a “Hawaiian Place of Learning.... more The University of Hawai‘i at Mānoa’s strategic plan includes being a “Hawaiian Place of Learning.” In this presentation we describe our longitudinal study (2010-2014) that captured student perceptions related to Hawaiian Place of Learning (HPL). We present results such as students’ HPL definitions, HPL’s importance to them, extent to which they view the university as a HPL, and their self-reports of amount learned about Native Hawaiian culture and where that learning took place. In addition, we discuss the strengths and challenges of a longitudinal research design and next steps regarding use of results.
This study explored college student perceptions of racial and ethnic stereotype-based humor in Ha... more This study explored college student perceptions of racial and ethnic stereotype-based humor in Hawai’i where humor is salient to the local culture. It has been suggested elsewhere that perceptions may be understood through eliciting a cultural domain or mental category including a set of items that are perceived to belong in the same category. For the present study, 72 students participated in interviews or focus groups to explore characterizations of racial and ethnic stereotype-based humor among university students in Hawai’i. The interviews consisted of generating free lists, rank orders and pile sorts, and were analyzed using ANTHROPAC software to convert textual data into matrices and cluster analyses to interpret the cultural domain. The findings indicated that students’ understanding of local stereotypes and humor aligned with the conceptualization of a social stratification in Hawai’i, because they acknowledged more derogatory stereotypes for certain racial and ethnic populations. As colleges continue to work towards diverse learning environments, humor is likely to be one of the more complex areas to address. Future research may involve a content analysis that examines campus fliers, events and media to understand better the role of the institution in developing and perpetuating existing stereotypes.
Uploads
Talks
The presenter will begin by briefly describing the purpose of using phone surveys (i.e., when to use phone surveys versus other data collection methods) and its advantages and disadvantages. Attendees will then be guided through a step-by-step process of how to (1) develop an effective phone survey (e.g., survey software, questions, length, flexibility, etc.), (2) establish a script or protocol to use when speaking to respondents, (3) create a tracking sheet to track respondents, and (4) appropriately communicate with respondents while conducting the survey (e.g., voice, tone, and persona, being adaptable, probing and clarification, etc.)
Throughout the demonstration, practical tips and advice that address ongoing challenges of phone survey research such as response rates, call scheduling, and reliability and validity will be shared. The presenter will also suggest techniques for addressing new and emerging challenges such as issues with caller ID, voicemail, privacy and confidentiality, and respondent burden.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
Beyer, C. H., Gillmore, G. M., & Fisher, A. T. (2007). Inside the Undergraduate Experience: The University of Washington's Study of Undergraduate Learning (JB-Anker Series).
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). K. A. Feldman (Ed.). San Francisco: Jossey-Bass.
Papers
The presenter will begin by briefly describing the purpose of using phone surveys (i.e., when to use phone surveys versus other data collection methods) and its advantages and disadvantages. Attendees will then be guided through a step-by-step process of how to (1) develop an effective phone survey (e.g., survey software, questions, length, flexibility, etc.), (2) establish a script or protocol to use when speaking to respondents, (3) create a tracking sheet to track respondents, and (4) appropriately communicate with respondents while conducting the survey (e.g., voice, tone, and persona, being adaptable, probing and clarification, etc.)
Throughout the demonstration, practical tips and advice that address ongoing challenges of phone survey research such as response rates, call scheduling, and reliability and validity will be shared. The presenter will also suggest techniques for addressing new and emerging challenges such as issues with caller ID, voicemail, privacy and confidentiality, and respondent burden.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
Beyer, C. H., Gillmore, G. M., & Fisher, A. T. (2007). Inside the Undergraduate Experience: The University of Washington's Study of Undergraduate Learning (JB-Anker Series).
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). K. A. Feldman (Ed.). San Francisco: Jossey-Bass.