Improving Alignment
Between Postsecondary
and Secondary Education:
The Texas College
and Career Readiness Initiative
American Educational Research Association
April, 2010
David T. Conley, PhD
Evelyn Hiatt
Charis McGaughy, PhD
Mary Seburn, PhD
Andrea Venezia, PhD
Contents
Symposium Overview....................................................................................................... 4
Conceptual Framework for the Symposium ................................................................ 5
Scientific or Scholarly Significance of the Studies ........................................................ 6
Project #1:
Development of the Texas College and Career Readiness Standards (CCRS).................... 8
Overview ................................................................................................................... 8
Methodology .............................................................................................................. 9
Results...................................................................................................................... 10
Significance .............................................................................................................. 11
Project #2:
Validation of the Texas College and Career Readiness Standards (CCRS) Against Current
practice in Entry-Level College Courses ......................................................................... 14
Overview ................................................................................................................. 14
Methodology ............................................................................................................ 15
Results...................................................................................................................... 20
Significance .............................................................................................................. 22
Project #3:
Creation of Reference Course Profiles in Entry-Level College Courses ........................... 23
Overview ................................................................................................................. 23
Methodology ............................................................................................................ 24
Results...................................................................................................................... 27
Reference Course Profile Template ........................................................................... 28
Significance .............................................................................................................. 33
Project #4:
Alignment Between the CCRS and Expectations in Entry-Level Career and Technical
Education Courses .......................................................................................................... 35
Overview ................................................................................................................. 35
Methodology ............................................................................................................ 37
Results...................................................................................................................... 42
Significance .............................................................................................................. 45
Project #5:
Alignment Between the CCRS and Placement Tests Commonly Used in Texas
Postsecondary Institutions.............................................................................................. 47
Overview ................................................................................................................. 47
Improving Alignment Between Postsecondary and Secondary Education
2
Methodology ............................................................................................................ 49
Results...................................................................................................................... 51
Findings.................................................................................................................... 53
Interpretation of Findings.......................................................................................... 73
References ...................................................................................................................... 78
Appendix A..................................................................................................................... 81
Appendix B..................................................................................................................... 89
Appendix C..................................................................................................................... 91
Appendix D .................................................................................................................. 103
Improving Alignment Between Postsecondary and Secondary Education
3
Symposium Overview
The goal of the symposium is to present findings from five separate investigations
undertaken as part of the Texas College and Career Readiness Initiative (TCCRI), a
statewide legislatively mandated set of activities designed to increase the number of
students attending college and succeeding once they reach college. This symposium
highlights the dimension of the TCCRI that is focused on improving alignment between
high school and college by defining and operationalizing postsecondary expectations for
college readiness. These activities were undertaken to define what constitutes
readiness, validate the definition against current practice in entry-level college courses,
develop “reference courses” to serve as benchmarks for college expectations, explore
the relationship between college readiness in key cross-disciplinary skills and the
expectations present in two-year certificate courses, and analyze the degree to which
placement tests measure the knowledge and skills defined as college ready. The
session will consider findings from these five aspects of the Initiative within a broader
policy context in order to illustrate significance both at the level of each study
individually and for the Initiative collectively.
Texas has been a leader in the movement to align high-school curriculum and
assessments with college readiness expectations, beginning with the initiation of
Closing the Gaps, a plan aimed to ensure high school students in Texas were prepared
for college and the workforce (THECB, 2009). In May 2006, the Texas Legislature
passed House Bill 1, a major piece of legislation that included multiple initiatives related
to high school success and college readiness. Its primary goal is to increase the number
of Texas high school students who graduate ready to succeed in college and 21st
Century careers.
The TCCRI represents a significant advancement in the field of systems alignment for
college readiness. No other state has undertaken such a comprehensive approach to
identifying, validating, and implementing the knowledge and skills necessary for college
Improving Alignment Between Postsecondary and Secondary Education
4
success. The Initiative consists of four phases that address a range of issues and
policies designed to improve statewide alignment between secondary and
postsecondary systems. Findings from activities undertaken in Phases I-III of the TCCRI
are presented in this symposium.
Conceptual Framework for the Symposium
Preparing all students for college and career readiness is a growing national priority as
increasing numbers of parents, educators, business leaders, and politicians emphasize
the importance of a highly educated workforce and citizenry (Achieve, 2006). Reaching
this difficult goal is complicated by a number of factors. These include the separate
governance systems for K-12 and postsecondary education (Conley, 2003), the lack of
shared information and expectations across educational systems, the limitations of the
existing means for determining college eligibility (Conley, 2007), the changes in the
demographics of American public school students (National Center for Educational
Statistics, 2007), and even the well-intentioned efforts of states to institute exit
examinations (Brown & Conley, 2007).
In order to prepare students to succeed in the U.S. economy and society, states have
adopted a strategy of aligning educational expectations vertically and holding schools
accountable for achieving defined outcomes (Susan H. Fuhrman, 1993; S.H. Fuhrman &
Elmore, 2004). For essentially all states, this means alignment of standards and
assessments across elementary, middle, and high school. The key missing link is
alignment of expectations between secondary and postsecondary education (Conley,
2005). As a concept, alignment is the underlying driver behind a range of state and
federal policies heretofore confined to K-12 education and only now being considered
seriously in relation to the transition from high school to college.
The symposium capitalizes on the systemic efforts by Texas to align postsecondary and
secondary education, using it as a framework to present a series of findings from an
interrelated set of activities designed to set the stage for more concrete actions to align
Improving Alignment Between Postsecondary and Secondary Education
5
high school and college. The intent is to balance each presentation between an
overview of each study and its findings and a discussion of the overall implications for
policy and practice. The studies included in this symposium include:
1. Development of the Texas College and Career Readiness Standards (CCRS)
2. Validation of the Texas College and Career Readiness Standards Against
Current practice in Entry-Level College Courses
3. Creation of “reference courses” in each of 20 entry-level college course subject
area
4. Analysis of alignment between the CCRS cross-disciplinary standards and the
expectations present in entry-level courses leading to two-year certificates at
public postsecondary institutions in Texas
5. Analysis of alignment between the CCRS and placement tests commonly used in
Texas postsecondary institutions
Scientific or Scholarly Significance of the Studies
The findings from these studies are important for a number of reasons, most important
among them is how they illustrate a stateʼs attempt to improve high school-college
alignment at the systems level and how such an effort can serve to inform other states.
Given current interest in “common core standards,” by the US Department of Education
and most states, a presentation that highlights postsecondary efforts to set collegeready standards and to align entry-level expectations to such standards will be highly
germane. If and when common core standards are adopted by states, the stage will be
set for discussions about the alignment of such standards with postsecondary
readiness. Having an example of one stateʼs efforts to anticipate the need to align high
school and college expectations will be useful and important.
Beyond the general findings from the five presentations, the symposium serves as a
sort of proof of concept for the notion of state-based initiatives to align high school and
college. The papers offer insights into the process and methodological issues
associated with an initiative issue of this sort and also highlight important relationships
Improving Alignment Between Postsecondary and Secondary Education
6
between college readiness standards and other aspects of the postsecondary system,
such as entry-level courses and placement tests. These insights can help inform policy
leaders nationally as they begin to grapple with alignment issues in earnest in the wake
of national efforts to focus high school standards on college and career readiness.
Finally, the results shed needed light on the methodological issues associated with
alignment work of this nature, which is becoming increasingly widespread and higher
stakes. Understanding how to go about establishing the content and expectations of
entry-level college courses will be an important area of investigation, and the
methodological strategies to do so will be of increasing interest. This symposium will
showcase a range of strategies and techniques for working in this area.
Improving Alignment Between Postsecondary and Secondary Education
7
Project #1:
Development of the Texas College and Career Readiness
Standards (CCRS)
Overview
In May 2006, the 79th Texas Legislature (Third Called Session) passed House Bill 1, a
major piece of legislation that included multiple initiatives related to high school success
and college and career readiness. This legislation added Section 28.008, entitled
“Advancement of College Readiness in Curriculum,” to Chapter 28 of the Texas
Education Code. Its goal was to increase the number of students who graduate from
Texas high schools ready to succeed in college and 21st Century careers.
In response to elements of this legislation, the Educational Policy Improvement Center
(EPIC) was awarded a contract issued by the Texas Higher Education Coordinating
Board (THECB) to facilitate the development and implementation of college and career
readiness standards as part of the Texas College and Career Readiness Initiative
(TCCRI). The purpose of the TCCRI is to develop and implement college and career
readiness standards and activities to improve alignment between secondary and
postsecondary education, resulting in an increased number of students prepared for
college and career success.
The TCCRI represents a significant advancement in the field of college and career
readiness. No other state has undertaken such a comprehensive approach to
identifying, validating, and implementing the knowledge and skills necessary for college
and career success.
In 2007, EPIC facilitated the development of the College and Career Readiness
Standards (CCRS), in partnership with the THECB and the Texas Education Agency
(TEA). Vertical teams composed of secondary and postsecondary instructors
representing all regions of the state engaged in the development process. These
Improving Alignment Between Postsecondary and Secondary Education
8
standards were adopted by the THECB in January 2008, and were sent to the
Commissioner of Education and State Board of Education for incorporation into the
Texas Essential Knowledge and Skills (TEKS) in April 2008. For details, please see:
Texas College and Career Readiness Standards, available online at:
http://www.thecb.state.tx.us/collegereadiness/TCRS.cfm.
The purpose of this central aspect of the TCCRI was to develop a set of standards that
represent what students must know and be able to do to succeed in entry-level college
courses in English/language arts, mathematics, science, and social studies. In addition
to content specification by subject area, they contain a stand-alone section that
specifies cross-disciplinary skills. These skills include many cognitive strategies and
techniques necessary for success in a wide range of college courses.
Methodology
The method used to generate the standards was a form of criterion-based expert
judgment that combined convergent consensus with theme analysis. Expert teams
comprised of 39 carefully selected and vetted secondary and postsecondary faculty
members met four times over a period of eight months, reviewed relevant extant
standards in their subject area and then completed online exercises between meetings.
The postsecondary faculty all taught entry-level courses. The secondary educators all
had expertise or experience developing content standards. Drafts were reviewed by
national experts and were made available for public comment. Results from the reviews
were incorporated into subsequent drafts.
The teams were instructed to use as their reference point the knowledge and skills a
student would need to be ready to succeed in a credit-bearing entry-level general
education college course at a state postsecondary institution. “Succeed” was defined as
being able to complete the entry-level course at a level that would allow further study in
the subject area, if the student chose to do so.
Improving Alignment Between Postsecondary and Secondary Education
9
Between meetings, EPIC analysts conducted theme analysis on input gathered at each
meeting and developed online draft versions that participants reviewed and commented
upon between meetings. In this fashion, numerous successive versions came to
approximate better the consensus view of the participants.
Results
The final version specified, by subject, the major standards, enduring understandings,
key concepts, performance expectations, and performance indicators necessary for
college success in each of the subject areas, including English/language arts,
mathematics, science, and social studies. In addition, a separate section, the crossdisciplinary standards, detailed the key cognitive skills and foundational skills necessary
for college and career readiness that transcend subject matter. The four nested levels
are organized as follows:
I. Key Content – Keystone ideas of a discipline that reverberate as themes
throughout the curriculum (designated by Roman numerals). Example: I. Numeric
Reasoning
A. Organizing Component – Knowledge and subject areas that organize a
discipline around what students should retain, be able to transfer, and apply to
new knowledge and skills (designated by capital letters). Example: A. Number
representation
1. Performance Expectation – Knowledge and skills that represent
important ideas of the current understanding of each organizing concept as
well as the multiple contexts in which each organizing concept can be
manifest (designated by numbers). Example: 1. Compare real numbers.
a. Performance Indicator – examples of how to assess and measure
performance expectations. This is not intended to be an exhaustive or
prescriptive list. The operating premise is that the more of these or other
similar indicators a student is successfully able to demonstrate, the greater
the probablilty that the student will be prepared to succeed in college
(designated by lowercase letters). Example: a. Classify numbers as
natural, whole, integers, rational, irrational, real, imaginary, and/or
complex.
Improving Alignment Between Postsecondary and Secondary Education
10
The resulting standards were made available for public review, and Texas
postsecondary faculty in particular were encouraged to review and comment upon the
draft. Overall, almost 1,000 individuals submitted comments. Table 1 lists the number of
participants submitting comments by subject area.
Table 1: Participation in Public Comment Process by Subject Area
Subject
Number of Individuals Submitting Comments
English
271
Math
197
Science
173
Social Studies
165
Cross-Disciplinary
165
Total
971
The comments were then analyzed and reviewed by the VT members as they prepared
their final drafts. The final version of the standards was submitted to the Commissioner
of Higher Education, who presented them to the THECB for adoption at its January
2008 meeting. The CCRS were approved unanimously (the THECB adopted the first
three levels; the fourth level, Performance Indicators, are intended to serve only as
examples) and sent to the Commissioner of Education and the State Board of Education
for incorporation into the Texas Essential Knowledge and Skills (TEKS).
Significance
The Texas College and Career Readiness Standards are significant for several reasons.
The method used to develop the standards moved beyond the traditional professional
judgment approach by triangulating current entry-level college and secondary faculty
membersʼ knowledge and expertise using an iterative process comprised of in-person
and online components, incorporating public comments, and validating the standards
against practices in entry-level college courses to confirm they do represent the
knowledge and skills necessary for postsecondary success. In addition, the VT
members used research and standards developed over the past 20 years in the relevant
Improving Alignment Between Postsecondary and Secondary Education
11
subject areas, using them as the foundation for creating the CCRS.
Whereas the CCRS serve as an important advancement in standards development,
they do differ fundamentally from traditional state high school graduation standards.
Overall, they have two different reference points. High school graduation standards tend
to focus on graduation (obtaining a high school diploma) as the terminal reference point.
They focus more on content knowledge and are the basis for state testing or
accountability requirements. The CCRS do not specify what students must master to
graduate from high school; rather, they set out what students need to know in order to
have a reasonable probability of success in their introductory college course at two- and
four-year institutions. The operating premise is that the more of these standards a
student can successfully demonstrate, the higher the probability for student success in
entry-level coursework and the avoidance of remedial placements. In addition, the two
sets of standards can complement each other: most state standards systems do not
continue to twelfth grand currently, ending at tenth or sometimes eleventh grade,
making the alignment to college readiness standards a reasonable exit level high school
extension.
The CCRS also contain a significant distinguishing feature and an important dimension
of college and career readiness: the identification and inclusion of the cross-disciplinary
standards. The necessity for including the cross-disciplinary standards emerged as the
Vertical Teams began to identify important knowledge and skills that transcended their
particular subject areas. For example, the Science and Mathematics Vertical Teams
collaborated to ensure that the standards in both areas addressed the mathematical
skills necessary to be successful in entry-level college sciences courses. Through these
cross-team conversations, the need for the cross-disciplinary standards became
apparent. These standards represent elements of learning that cut across disciplines,
and are organized into two areas: Key Cognitive Skills and Foundational Skills. The Key
Cognitive Skills specify intellectual behaviors that are prevalent in entry-level college
courses (such as intellectual curiosity, reasoning, problem solving, academic behaviors,
Improving Alignment Between Postsecondary and Secondary Education
12
work habits, and academic integrity). Foundational Skills consist of proficiencies
students need to be able to transfer knowledge and apply it across the curriculum (such
as reading, writing, conducting research, understanding and using data, and using
technology). One of the strongest findings from the subsequent validity studies of the
CCRS (reported throughout the remainder of the this symposium) is the high level of
alignment of the cross-disciplinary standards have with faculty expectations for
readiness in entry-level college courses in both general education and Career and
Technical Education courses. Through the identification and validation of the high level
of importance for success in entry-level college work, the cross-disciplinary skills
represent an important body of knowledge all educators are responsible for teaching
within their own discipline and context.
The CCRS also represent a unique state-level effort to connect a stateʼs K-12 standards
with college and career readiness. The stateʼs willingness to undertake significant
legislatively-sponsored statewide alignment activities could serve as a model for other
states. These standards serve as the basis for multiple alignment activities. They can be
compared to the state high school standards and assessments as a means to
determine how well the two are aligned. In addition, districts and high schools can use
the standards to align their course expectations more directly with college readiness.
This is particularly important for courses beyond the level of or content not covered in
the state high school examinations that currently have no explicit reference points.
Overall, Texas is an important state in its ability to influence textbook publishers, which
could lead to the standards influencing curriculum and textbooks in other states. The
CCRS also serve as an important reference point for the higher education system,
enabling additional alignment activities such as departmental internal and external
course alignment self-examinations, and explorations of placement tests.
Improving Alignment Between Postsecondary and Secondary Education
13
Project #2:
Validation of the Texas College and Career Readiness
Standards (CCRS) Against Current practice in Entry-Level
College Courses
Overview
The College and Career Readiness Standards (CCRS) were then validated against
current practice by analyzing the content of over 930 syllabi from 20 core entry-level
general education courses taught at two- and four-year postsecondary institutions
throughout the state. The complete findings from this research are available in the
report Examining the Alignment between the Texas College and Career Readiness
Standards and Entry-Level College Courses at Texas Postsecondary Institutions
(Referred to hereafter as the TCCRI Phase II Validity Study) released by EPIC in
October 2008.
During Phase I of this project, completed in January 2008, EPIC facilitated the
development of the CCRS, in partnership with the THECB and the Texas Education
Agency (TEA). Vertical teams composed of secondary and postsecondary instructors
representing all regions of the state engaged in the development process. These
standards were adopted by the THECB in January 2008, and were sent to the
Commissioner of Education and State Board of Education for incorporation into the
Texas Essential Knowledge and Skills (TEKS) in April 2008.
The next step in the process was to conduct an alignment analysis to compare the
standards to what is actually being taught in entry-level college courses in
postsecondary institutions throughout Texas. The purpose was to establish the validity
of the standards as an accurate representation of the key knowledge and skills
necessary for college and career readiness and success. The results of the study can
be used to affirm the accuracy of elements of the CCRS and to identify areas where
additions, deletions, or modifications to the standards should be considered.
Improving Alignment Between Postsecondary and Secondary Education
14
Methodology
This study was designed to answer the following question: How do the CCRS compare
to what is currently taught in entry-level college courses at Texas institutions of higher
education? This question was addressed through a validity analysis identifying the
alignment and any gaps between the CCRS and current instructional content and
practices in 20 entry-level college courses. Over 800 entry-level college course
instructors in eight subject areas representing 20 separate course titles rated the
importance of what they taught in those courses in relation to the CCRS. They also
submitted their course syllabi. External reviewers analyzed the match between the
instructor ratings of importance and the content of their syllabi to gauge the accuracy of
the self-ratings.
To start this review process, the Commissioner of Higher Education asked the Chief
Academic Officers of all state and private colleges and universities in Texas (with the
exception of 12 health-related and private career schools) to designate a College
Readiness Special Advisor (CRSA) at their respective campuses. The THECB invited
138 public and private institutions of higher education to designate a college readiness
special advisor; 108 institutions accepted the invitation. Two public and 28 private
institutions did not respond to the invitation.
The THECB charged the CRSAs with soliciting course nominations from their respective
institutions. To select the 20 course titles for which data would be collected, the THECB
identified courses with high enrollment statewide among entry-level students along with
“gatekeeper” courses that are prerequisite requirements for different majors. The
THECB selected courses according to the Texas Common Course Numbering System,
a uniform set of course designations that the majority of Texas institutions of higher
education employ to help facilitate the transfer of entry-level courses between
institutions. The course titles selected include the following:
•
BIOL 1406 - Biology for Science Majors I
Improving Alignment Between Postsecondary and Secondary Education
15
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
BIOL 1408 - Biology for Non-Majors I (lecture & lab)
BIOL 2401 - Anatomy and Physiology I
CHEM 1405 - Introductory Chemistry I
CHEM 1412 - General Chemistry II (lecture & lab)
ENGL 1301 - Composition I
ENGL 1302 - Composition II
ENGL 2332 - World Literature I
GOVT 2301 - American Government I
GOVT 2302 - American Government II
GOVT 2305 - Federal Government
GOVT 2306 - Texas Government
HIST 1301 - U. S. History I
HIST 1302 - U. S. History II
MATH 1314 - College Algebra
MATH 1324 - Mathematics for Business & Social Sciences I
MATH 1342 - Elementary Statistical Methods
PHYS 1401 - College Physics I
PHYS 1405 - Elementary Physics I
PSYC 2301 - General Psychology
Between October 2007 and January 2008, Course nominations were collected from
CRSAs via a web-based application designed for this specific purpose. Nominations
included courses that were considered exemplary, were aligned with the CCRS, or that
stood out in some other way as the best representation of an entry-level course in the
course title area. The special advisors nominated 1,211 courses by submitting the
faculty memberʼs name and contact information and the institution-specific course title
when appropriate. The distribution of nominated courses reflected relatively closely the
distribution of the state public two-year and four-year institutions by geographic area.
In March 2008, instructors whose courses had been nominated received an email
notifying them of the availability of the course submission site. Submitting faculty
members provided information on course objectives, class size, grading policy, texts
used, prerequisite and pathways courses, and percent of students who enter their
course well prepared. Instructors were then instructed to upload an electronic version of
the course syllabus. Table 2 reports the distribution of all course submissions
Improving Alignment Between Postsecondary and Secondary Education
16
(completed and partial) by region and institution type. In total, EPIC collected 960
course submissions, including 913 complete submissions and 47 partial.
Table 2: Distribution of all Course Submissions by Region and Institution Type
Community
College
4 Year Public
University
Technical
College
Private
College
Total
Central
58
33
3
1
95
Gulf Coast
93
63
0
0
156
High Plains
35
21
0
0
56
Metroplex
126
54
0
0
180
Northwest
44
16
3
0
63
South
87
69
14
1
171
Southeast
11
29
31
0
71
Upper East
58
16
4
6
84
Upper Rio Grande
17
10
0
0
27
West
30
27
0
0
57
Total
559 (58%)
338 (35%)
55 (6%)
8 (1%)
960
Region
After providing course information and submitting a syllabus, instructors completed an
online rating form that asked them to answer the following question for each relevant
content area standard and cross-disciplinary standard: “How necessary is this element
in preparing students to succeed in my course?” Respondents chose one of five
options: Most necessary, More necessary, Less necessary, Least necessary, or Not
necessary. Instructors also had the option of skipping any section of the standards if
that section was not relevant to their course (e.g., biology instructors could skip the
physics section). After selecting a response option for each standard, instructors then
chose one or more rationale statements to explain their reasoning. (See Appendix B for
a list of scale items and rationale statements.) The rationale statements were designed
to provide greater clarity to understand the ratings. For example, an instructor might
designate a standard as Not necessary or Least necessary for one of several reasons.
The standard could not be necessary to succeed in the course because it was irrelevant
to the subject area, or because it would be covered in a subsequent course. Distinctions
of this nature are important when considering discrepant standards. Including the
Improving Alignment Between Postsecondary and Secondary Education
17
rationale statements were particularly valuable in interpreting the reasons why specific
standards were not well aligned.
External Review to Check the Reliability of Self-Ratings
A common criticism of self-ratings is the potential lack of reliability of such ratings. Selfratings have the advantage of generating a tremendous amount of information in an
efficient and cost-effective manner. However, such ratings should be verified so that
findings based on the ratings can be interpreted with confidence as being representative
of the actual state of the phenomenon being investigated, in this case the importance of
each CCRS in preparing students for success in entry-level college courses.
The method selected to examine the accuracy of the self-ratings was a concurrent
expert review. This method uses trained subject-matter experts to use course syllabi
from each submitted course to verify the assertions made by the instructor of that
course on each CCRS. In other words, if an instructor stated that a particular CCRS was
Most important for success in a course, the external reviewer would examine the course
syllabus to determine whether it contained evidence to support the instructorʼs selfratings of alignment with the CCRS. The expert reviewer could choose among three
statements: Evident, Not evident, or Not applicable. By selecting Evident, the expert
reviewer indicated that the standard was Reasonable to infer, Stated verbatim, or
Implied in the course documents.
Expert reviewers examined a sample consisting of approximately 20 percent of courses
submitted in each course title area. Table 2 presents the percent of instructor responses
confirmed by external reviewers by course title. The rate of confirmation ranges from 96
percent in BIOL 2401 (Anatomy and Physiology I) to 50 percent in CHEM 1405
(Introductory Chemistry I).
Improving Alignment Between Postsecondary and Secondary Education
18
Table 2: Percent of Instructor Responses Confirmed by Course Title
# of Courses
Reviewed
% of Instructor Responses
Confirmed
BIOL 1406 - Biology for Science Majors I
11
89%
BIOL 1408 - Biology for Non-Majors I (lecture & lab)
6
86%
BIOL 2401 - Anatomy and Physiology I
9
96%
CHEM 1405 - Intro Chemistry I
9
50%
CHEM 1412 - General Chemistry II (lecture & lab)
8
60%
ENGL 1301 - Composition I
13
90%
ENGL 1302 - Composition II
12
94%
ENGL 2332 - World Literature I
11
92%
GOVT 2301* - American Government I
8
89%
GOVT 2302* - American Government II
6
83%
GOVT 2305* - Federal Government
4
77%
GOVT 2306* - Texas Government
4
91%
HIST 1301 - U. S. History I
11
85%
HIST 1302 - U. S. History II
11
79%
MATH 1314 - College Algebra
15
88%
MATH 1324 - Mathematics for Business & Social Sciences I
12
92%
MATH 1342 - Elementary Statistical Methods
11
87%
PHYS 1401 - College Physics I
9
63%
PHYS 1405 - Elementary Physics I
4
68%
PSYC 2301 - General Psychology
10
53%
Course Title
Results indicate acceptable to very high matches between instructor self-ratings of
courses against the CCRS and the expert reviewer confirmation of the ratings. Only five
courses fell below the 75 percent level of verification. Given the inherent difficulties of
determining instructor intent from a syllabus and the complexity and number of the
College and Career Readiness Standards, the levels at which external reviewers found
evidence of the CCRS within the syllabi chosen for review is sufficiently high to have
confidence that the instructor self-ratings can be taken as accurate representations of
course content.
Determining Alignment Thresholds
Once all data from instructors were received and analyzed, it was possible to establish
the criterion point for determining that alignment existed or did not exist between the
Improving Alignment Between Postsecondary and Secondary Education
19
CCRS and the course content. In essence, this was a standard setting activity in its own
right. For each individual CCRS statement, the modal instructor response was
determined. Each standard could have a score of 1 through 5, representing each of the
response options. The modes for each CCRS statement were examined via a scatter
plot in order to identify overall patterns. In some cases, a CCRS statement had multiple
modes. This analysis confirmed the appropriateness of collapsing the categories of
Most necessary and More necessary, using this new category to identify alignment
between the CCRS statement and instructor self-rating of CCRS importance. CCRS
statements for which the modal score was Less necessary were labeled as
inconsistently aligned, and CCRS statements that had a modal score of Least
necessary or Not necessary were identified as being not aligned.
Results
Ultimately, over 800 instructors representing 87 IHE and four college districts submitted
over 900 syllabi and ratings from 20 different entry-level, credit-bearing courses. Results
from the analysis revealed that the CCRS are highly aligned with entry-level college
courses in Texas. Rates of alignment by subject area for all standards were 99 percent
in social studies, 97 percent in English/language arts, 87 percent in mathematics, and
86 percent in science. For the cross-disciplinary standards, 100 percent are aligned
across the four subject areas (90 percent are aligned within each of the four subject
areas individually). Whereas all of the CCRS may not be aligned in any single course,
an examination across all courses within a given subject area reveals the high degree of
alignment between the CCRS and all entry-level courses in that subject. Table 3 below
lists the results by subject area, and also includes the overall levels of alignment for the
cross-disciplinary skills both aggregated and by subject area.
Improving Alignment Between Postsecondary and Secondary Education
20
Table 3: Summary of Overall Alignment by Subject Area and Cross-disciplinary Skills
Aligned
Inconsistently
Aligned
Not Aligned
Multimodal
All English
97%
2%
1%
0%
All Math
87%
8%
5%
0%
All Science
86%
14%
0%
All Social Studies
99%
1%
0%
0%
Cross-Disciplinary
All Subjects
100%
0%
0%
0%
Cross-Disciplinary
All English
98%
0%
2%
0%
Cross-Disciplinary
All Math
91%
9%
0%
0%
Cross-Disciplinary
All Science
100%
0%
0%
0%
Cross-Disciplinary
All Social Studies
98%
2%
0%
0%
Subject
Note: for a detailed exploration of the alignment results at the individual standard level within each subject area and
course, please refer to the original TCCRI Phase II Validity Study.
Two notable exceptions to the overall high level of alignment emerged in this study.
First, three key content areas were found to be not well aligned: Geometric Reasoning
in mathematics; and Earth and Space Sciences and Environmental Science in science.
The reason for the lower levels of alignment in these three areas is due to the fact none
of the entry-level courses selected for inclusion in the data collection for this study
typically cover that content, and an additional study would be necessary to analyze the
relationship between those standards and the appropriate entry-level courses not
included in this study. Therefore, the overall rates of alignment for all standards in
mathematics and science alignment understate the actual degree of alignment by
including standards in these three areas not included in the data collection. The
adjusted rates of alignment for the standards represented in the data collection are 93
percent for mathematics and 92 percent for science.
Secondly, there was a notably low level of alignment between the physics CCRS and
Improving Alignment Between Postsecondary and Secondary Education
21
current practice in entry-level physics courses. Only 56 percent of the 50 physics
standards were rated most necessary or more necessary for preparation for success in
entry-level physics courses, compared to 91 percent in biology and 94 percent in
chemistry. Further analysis of the rationale statements provided by the physics
instructors found the primary reason that the physics standards were not rated as
aligned was because the faculty taught those standards as new material at the college
level. When presented with these findings, the original CCRS Science Vertical Team
recommended the physics standards to remain unchanged due to the nature of the
discrepancy. Having the opportunity to learn the physics standards in the secondary
setting would only help better prepare students for what the postsecondary faculty will
be teaching in their entry-level physics courses.
Significance
This study is important because it is the first large-scale validity study of college
readiness standards undertaken at a state level. Previous college readiness standards
were developed by expert judgment panels and were often buttressed by exemplar
materials. But none of these went so far as to validate the standards against current
practice in college classrooms by using instructor review and syllabus analysis to
confirm the expert judgment process. This study offers insight into the methods for
conducting a validation study of this nature in addition to presenting the empirical
findings specific to the CCRS. It is important to have validated college readiness
standards particularly in states that are considering aligning high school and college
expectations, standards, or assessments. The validated college readiness standards
can then be used for planning purposes between state K-12 education departments and
postsecondary system offices. Given the high degree of investment that education
departments in most states have in the stateʼs high school standards and assessments,
it is unlikely most states will make changes in these standards to improve system
alignment unless it can be proven that the college readiness standards are a valid
representation of current practice in the stateʼs entry-level college courses.
Improving Alignment Between Postsecondary and Secondary Education
22
Project #3:
Creation of Reference Course Profiles in Entry-Level
College Courses
Overview
A separate component of the Phase II Study analyzed the results of the course ratings
to identify common practice within current entry-level courses and develop 18 Reference
Course Profiles. The Reference Course Profiles are composite courses designed to
represent the content and rigor of what is typically being taught currently in entry-level
college courses, and provide a snapshot of current practice (not best practice). A profile
includes a course syllabus with significant additional detail along with attendant course
materials, such as assignments, assessments, and scoring rubrics. The reference
courses do not represent best practice; instead, each of the 18 reference courses was
intended to represent normative practice at a wide range of postsecondary institutions
within Texas. While in practice any individual course at a postsecondary institution might
differ from the reference course, enough consistency will exist for the reference course
to be a useful target for secondary teachers to use in preparing students for college. A
more detailed description of this research can be found in the last section of the report
Examining the Alignment between the Texas College and Career Readiness Standards
and Entry-Level College Courses at Texas Postsecondary Institutions (or the TCCRI
Phase II Validity Study) released by EPIC in October 2008.
The purposes of the Reference Courses are two-fold. At the secondary level, instructors
can refer to the materials as they prepare their students for the course content they will
encounter when they reach college. At the postsecondary level, the materials serve as a
point of comparison that faculty can use when creating or refining entry-level courses.
Whereas the use of the Reference Course Profiles is purely voluntary, the goal for
institutions of higher education is to ensure that entry-level courses are aligned with the
CCRS, contain college-level content, and are cognitively challenging. By making
Improving Alignment Between Postsecondary and Secondary Education
23
expectations more transparent, the Reference Course Profiles will help students,
educators, and policymakers understand more clearly and reach agreement more
quickly on the nature of the student preparation necessary for college success.
Methodology
This study was designed to answer the following question: What are the most common
components of and current practices in entry-level college courses in Texas
representing 18 specific course titles? To answer this question, 18 Reference Course
Profiles were created through a linked criterion-based analytic process. In short, this
approach linked several steps including identifying the modal courses and designing the
profiles. The CCRS and the modal faculty expectations relative to the standards in
entry-level courses within a course title were used as the criteria throughout the
process. The modal course profile represents the typical instructor response pattern for
each course title. Using the modal course profile, 10 individual courses were identified
as the most highly aligned with the typical aggregate rating results for each course title.
The course materials from these modal courses paint a picture of the most frequent
current instructional priorities and practices for the specified courses in Texas
institutions of higher education.
The Reference Course Profiles were compiled from data collected from faculty ratings,
syllabi, and additional course documents submitted by instructors in entry-level college
courses throughout Texas during the data collection process described in the Phase II
Study in the previous section. Table 4 shows the distribution of submitted syllabi by
institution type. This table reports only usable submitted syllabi, and does not reflect the
number of instructor rating completions or partial submissions.
Table 4: Distribution of Submitted Syllabi by Institution Type
Syllabi Submitted:
2-year
Syllabi Submitted:
4-year
Syllabi Submitted:
Total
BIOL 1406
26
33
59
BIOL 1408
16
17
33
Course
Improving Alignment Between Postsecondary and Secondary Education
24
Syllabi Submitted:
2-year
Syllabi Submitted:
4-year
Syllabi Submitted:
Total
BIOL 2401
24
21
45
CHEM 1405
19
30
49
CHEM 1412
21
23
44
ENGL 1301
30
37
67
ENGL 1302
27
34
61
ENGL 2332
24
32
56
GOVT 2301
10
21
31
GOVT 2302
10
17
27
HIST 1301
19
31
50
HIST 1302
21
32
53
MATH 1314
28
40
68
MATH 1324
29
30
59
MATH 1342
21
32
53
PHYS 1401
21
27
48
PHYS 1405
11
11
22
PSYCH 2301
24
30
54
Total
381
498
879
Course
Creation of the Modal Courses
The first step in the linked criterion-based analytic process was to identify the subset of
courses most aligned with current practice in entry-level college courses in Texas. The
data analyzed consisted of a scale item response for each standards statement. Scale
item responses were represented by the numbers one through six and corresponded
with Not necessary, Least necessary, Less necessary, More necessary, Most
necessary, and Skip section, respectively. As in the alignment study, responses left
blank were treated as missing data and excluded from the analysis.
Then, a “modal course profile” was created that represented the typical response
pattern for the set of rated standards for the identified courses in each course title area,
18 in total. The modal course was determined by the aggregation of the results of the
instructorsʼ self-ratings representing the necessity of each CCRS for successful
preparation for the course. In other words, the modal course profile reflects the CCRS
as they are most typically represented in that course title area.
Improving Alignment Between Postsecondary and Secondary Education
25
Proximity Measures
Once the modal course profiles were generated, the next linked step was to calculate a
measure reflecting the percent agreement between the ratings for each course and the
modal course ratings (a percent match measure) along with three proximity measures
that examined the similarity between each of the selected courses in a course title area
and the modal course in that area. The proximity measures included: 1) a Euclidean
distance measure, 2) a measure of the correlation of the response pattern for each
course and the modal course, and 3) the cosine value of the angle between the
response pattern for each course and the response pattern for the modal course. These
three proximity measures, though distinct, resulted in course orderings that were highly
related to one another.
Next, a factor analysis considered all proximity measures as well as the percent match
value to compute a final score for each course using a linear combination of the four
similarity measures. These factor scores were used to rank-order the courses in order of
similarity to the modal course based on the combination of similarity measures. The ten
most similar courses for each course title then become the basis for building Reference
Course Profiles that integrate the practices found across these courses, paying special
attention to the most similar courses.
The ranking of these factor scores were used to rank-order a total of 180 courses
across the 18 course title areas, i.e., the ten courses most aligned to the modal course
for each of the 18 Reference Course Profiles titles. The distribution of these 180
courses is similar to the distribution of all courses submitted, although community and
technical colleges are slightly overrepresented, and four-year universities and private
colleges are slightly underrepresented. Private colleges are absent from the list of most
representative courses.
Additional materials, such as assignments and assessments, were collected from
instructors of these identified courses early in Phase III, beginning Fall 2008. Design
Improving Alignment Between Postsecondary and Secondary Education
26
teams created the Reference Course Profiles through a criterion-based professional
judgment process in which decisions about what to include in the profile are made
based on alignment with the CCRS and the content of the ten highly aligned courses per
course title area.
These additional course artifacts provided deeper insight into the ways in which
instructor expectations relate to the CCRS. Design teams were formed for each content
area, and each team was composed of three postsecondary instructors. These design
teams synthesized the results from the linked criterion-based analytic process to create
the Reference Course Profiles. The resulting Reference Course Profiles will then serve
to demonstrate how and why the relevant CCRS for that course title area are important
to success in the course.
Results
Design teams created the 18 Reference Course Profiles through a criterion-based
professional judgment process. Decisions about what to include in the Reference
Course Profile were made based on alignment with the CCRS and the course materials
from the ten modal (highly aligned) courses per course title area. The additional course
artifacts provided deeper insight into the ways in which instructor expectations relate to
the CCRS. Design teams were formed for each content area, and each team was
composed of three postsecondary instructors. These design teams synthesized the
results from the linked criterion-based analytic process to create the Reference Course
Profiles. The resulting Reference Course Profiles not only provide a snapshot of current
practice, but also demonstrate how and why the relevant CCRS for that course title area
are important to success in the course.
Improving Alignment Between Postsecondary and Secondary Education
27
Reference Course Profile Template
This next section outlines the template that was used by each of the design teams to
develop the Reference Course Profiles. The template includes all of the section
headings of the profile, and short descriptions. For a complete Reference Course Profile
example, please refer to “Appendix C: Reference Course Profile: English Composition
II.”
Course Name
Course Number
Course description appearing in the Lower Division Academic Course Guide Manual.
Reference Course Profile: Overview and Purpose
A Reference Course Profile represents current practice in entry-level college courses in
Texas. Twenty-six course profiles were developed from an extensive study of over 900
entry-level general education and CTE college courses offered at Texas institutions of
higher education. Each course profile consists of a course description, sample syllabus,
reference lists, attendant course materials (such as assignments, assessments, and, in some
cases, scoring rubrics), and the Texas College and Career Readiness Standards addressed
within the course.
The Profiles provide a reference point for college readiness that fosters increased
transparency between secondary and postsecondary education. They are not intended to
be mandatory, prescriptive, or best practice.
Prerequisites and Prior Knowledge
Most college-level syllabi do not list all prior knowledge required to succeed in the
course; it is usually an underlying assumption. For the purpose of this Reference Course
Profile, the required prior knowledge and skills students need to be successful in the
course are explicitly stated to help both secondary and postsecondary faculty establish
goals and expectations for their students. The knowledge and skills reflected in this section
are pulled directly from the Texas College and Career Readiness Standards (CCRS), written
and validated by Texas faculty during the 2007-8 school year. The CCRS are available
online at: http://www.thecb.state.tx.us/collegereadiness/CRS.pdf
Prior to enrolling in this course, students must satisfy Texas Success Initiative (TSI)
requirements set by the institution as described in Coordinating Board rule (Texas
Administrative Code, Chapter 4, Subchapter C).
An instructor might also require a pre-semester diagnostic test to help the student
assess his or her current ability in _____________.
Improving Alignment Between Postsecondary and Secondary Education
28
In addition, students should have the skills reflected in the following College and Career
Readiness Standards. Only the specific standards and performance expectations
pertinent to the course are listed below.
College and Career Readiness Standards
I.
A.
1.
2.
Cross-Disciplinary Standards
I. Key Cognitive Skills
II. Foundational Skills
Course Objectives
Course objectives include the course-specific skills and knowledge that students will
possess upon completion of the course. They assist postsecondary faculty in clarifying the
goals of their courses and provide a clear picture of the expectations students will
encounter once they begin college. This sample list of objectives was adapted from syllabi
submitted in 2008 by Texas college faculty.
In this course, [course title], students will learn________________________________.
Upon completion of the course the student will be able to:
1.
2.
Sample Textbooks and Materials
This list is comprised of texts that appear on course syllabi submitted in 2008 by faculty
teaching entry-level college courses most representative of current practice in Texas. This
list is not exhaustive, prescriptive, or required.
Sample Methods of Instruction
Students should be prepared to encounter a variety of instructional methods, as faculty
indicate the use of several beyond the lecture format. The list of methods of instruction has
been adapted from course syllabi submitted in 2008 by faculty teaching entry-level
college courses most representative of current practice in Texas. The approximate
percentage of time allocated to each instructional method is also indicated.
Improving Alignment Between Postsecondary and Secondary Education
29
1. Lecture – X%
a. Lecture is defined as __________________________________________.
b. Lectures will take place in the form of _____________________________.
2. Full-class Discussion – X%
a. Students will be expected to come to class ready to contribute to class
discussion.
b. Students will be expected to listen and respond to each otherʼs comments.
3. Method of Instruction – X%
Sample Assignments
A typical number of formal assignments for [course title] is _______. The course will also
include a number of other types of assignments including _______. Below are the kinds of
assignments that might be expected and the percentage of the final grade each might
carry.
1. Written Work – X%
a. Statement re: appropriate format.
b. Statement re: appropriate citation style.
c. Statement re: appropriate writing style.
d. Statement re: length requirement:
2. Other assignment type – X%
a. Statement of information necessary to inform student expectations.
3. Other assignment type – X%
a. Statement of information necessary to inform student expectations.
Sample Assessments
Typically [course title] will include _______ formal examinations. Typically the
culminating assessment for the course will be _______.
Specific Assessments:
1. Quizzes – X%
a. Statement of information necessary to inform student expectations.
2. Final Exam – X%
a. Statement of information necessary to inform student expectations.
Improving Alignment Between Postsecondary and Secondary Education
30
Sample Schedule
Samplings of assignments and assessments have been provided in the Reference Course
Profile materials. Bolded blue text indicates when a document is provided and a link to
that document is available. The time allocated for students to complete the assignment is
also indicated. The list of topics, as well as the overall pacing of the course, has been
adapted from course syllabi submitted in 2008 by faculty teaching entry-level college
courses most representative of current practice in Texas.
Week
Reading/Discussion Topics
Assignments & Assessments
1
Assignment 1:
2
Complete ______ exercise. Take ______ quiz.
3
Assignment 2:
4
Take ______ quizzes.
5
Assignment 3:
Complete ______ exercise. Take ______ quiz.
6
Take ______ quizzes.
7
Assignment 4:
Take ______ quizzes.
8
Exam 1 – Midterm Assessment
9
Assignment 5:
10
Take ______ quiz.
11
Assignment 6:
12
Complete ______ exercise. Take ______ quiz.
13
Assignment 7:
Class presentations
14
Assignment 8:
Final conferences
15
Exam 2 – Final Exam
Improving Alignment Between Postsecondary and Secondary Education
31
Sample Class Policies and Expectations
Students often enter college unaware of expectations regarding attendance, participation,
behavior, grading, and academic integrity. Faculty frequently include detailed policies and
expectations in their syllabi, making explicit to students the standards of successful
participation. Institutional-level policies are often included as well.
Attendance Policy
Students are expected to attend class regularly and to complete assignments on the
days specified.
Expectations
Student Conduct
• No _______________________ permitted in the classroom.
• Disruptive behavior will not be tolerated.
• Examples of disruptive behavior include _______________________.
Academic Behaviors
Students are expected to _______________________.
Grading Policy
Overview of Grades and Grading Standards
• A (90–100) = excellent/performance beyond mastery
• B (80–89) = above average/beyond basic mastery
• C (70–79) = average mastery
• D (60–69) = below average
• F (0–59) = failure
Percentages of Grade
• Assignment Type – X%
• Assessment Type – X%
• Quizzes – X%
Academic Integrity Policy
Sample List of Student Resources
The following list is representative of the resources mentioned in 2008 entry-level college
course syllabi in Texas. The resources listed are indicative of the expectation that
postsecondary students take responsibility for their own learning. As often explicitly stated
in course syllabi, students are expected to take advantage of these resources, and may
even be required to do so.
Improving Alignment Between Postsecondary and Secondary Education
32
Supplementary Documents
The section lists the additional documents included within the Reference Course Profile,
but not included in the Sample Schedule. These supplementary materials are intended to
enhance both instruction and student learning, and may be used at instructors’ discretion.
Significance
The current phase of the Texas College and Career Readiness Initiative uses the
Reference Course Profiles as a primary reference point in the creation of College
Readiness Assignments that are being designed to assess college readiness. College
Readiness Assignments, scoring guides, and work samples that align with the
Reference Course Profiles will be disseminated statewide as part of a professional
development effort to assist educators in improving student preparation for creditbearing entry-level college coursework and improving transparency across the
secondary and postsecondary systems.
The Reference Course Profiles are important tools for systems alignment because they
create an operational connection between the CCRS and actual practices in college
courses. They help move the alignment discussion beyond anecdote and assertion
regarding the nature of postsecondary expectations by offering concrete, specific
descriptions of college courses against which high school programs of study can be
aligned. For the first time, a state has sponsored a system wide analysis of entry-level
college courses, thereby providing a research-based point of reference.
An additional use of the profiles is as a resource for improvements to postsecondary
education. One of the results of the decentralization of postsecondary governance and
instruction is that it is essentially impossible to compare practices across a set of
courses with the same title offered at different institutions, or even at the same
institution. The Reference Course Profile is a potential tool for comparing the
expectations, content coverage, and challenge level present in any individual course to
a more independent standard. While the profile is not designed to standardize
instruction across institutions, nor represent best practice, it is a useful resource for
Improving Alignment Between Postsecondary and Secondary Education
33
helping to enhance consistency across entry-level courses, which can foster improved
alignment between secondary and postsecondary education, if secondary educators
have consistent targets toward which to align their courses and instruction.
Improving Alignment Between Postsecondary and Secondary Education
34
Project #4:
Alignment Between the CCRS and Expectations in EntryLevel Career and Technical Education Courses
Overview
In Fall 2008, EPIC was awarded an additional contract issued by the THECB to explore
the relationship between the cross-disciplinary standards and Career and Technical
Education (CTE) courses. Under this contract, EPIC replicated the validation
methodology of the TCCRI Phase II Validity Study, by comparing the CCRS crossdisciplinary standards to nine CTE courses at two-year institutions throughout the state.
This report summarizes the results from the study. The complete findings from this
research are available in the report Texas College and Career Readiness Initiative:
Texas Career and Technical Education Phase I Alignment Analysis Report (referred to
hereafter as the Phase I CTE Study) released by EPIC in March, 2009.
The purpose of this study was to examine the relationship between the Texas College
and Career Readiness Standards (CCRS) and what is currently being taught in nine
entry-level Career and Technical Education (CTE) courses throughout Texas. The
findings from this research will enable high school faculty to determine the degree to
which what they are teaching is aligned with the knowledge and skills necessary for
success in a community college or technical college environment. Furthermore, both
high school and postsecondary faculty teaching CTE courses will have a concrete
benchmark against which they can compare the challenge levels of their courses.
A unique feature of the CCRS is the inclusion of cross-disciplinary skills that span all
subject areas. These are the foundational cognitive skills that underlie and connect all
disciplinary areas and that students need to be able to apply across a variety of
contexts and subject matter. They relate to 21st Century learning and work environments
in which the cross-disciplinary skills are prerequisites to solving many of the most
important problems students will encounter in college and the workplace. The crossImproving Alignment Between Postsecondary and Secondary Education
35
disciplinary standards are divided into two areas: Key Cognitive Skills, such as
reasoning, problem solving and conducting research; and Foundational Skills to process
and create content knowledge, such as reading, writing, and data analysis.
The Phase I CTE Study consists of two phases designed to identify the relationship
between CTE courses and the cross-disciplinary CCRS, and then to illustrate those
relationships in more concrete fashion. The two phases are outlined below:
Phase I: Alignment Analysis
Nine entry-level CTE courses were analyzed against the CCRS cross-disciplinary
standards, which are arranged in four nested levels. The THECB adopted the
first three levels; the fourth level includes Performance Indicators intended to
serve only as examples. This study analyzed the first three levels, which
represent the Key Cognitive and Foundational Skills and knowledge necessary
for college and career preparation. The cross-disciplinary skills are organized as
follows:
I. Key Cognitive Skills – overarching skills that are necessary for
success across the curriculum. Example: I. Key Cognitive Skills
A. Organizing Component – cognitive skills for which a range of
specific strategies can be developed and mastered. Example: A.
Intellectual Curiosity
1. Performance Expectation – specific strategies that make
up the cognitive skill. Example: 1. Engage in scholarly inquiry
and dialogue.
a. Performance Indicator – examples of how to
assess and measure performance expectations. This
is not intended to be an exhaustive list. Example: a.
Identify what is known, not known, and what one
wants to know in a problem.
Improving Alignment Between Postsecondary and Secondary Education
36
Phase II: Reference Course Profile Development
Based on the course data collected and analyzed in Phase I, design teams made
up of postsecondary CTE content experts developed composite Reference
Course Profiles to represent the content and rigor of courses in which entering
college students should be prepared to succeed.
Methodology
This Phase I study was designed to answer the following question:
How do the standards contained in the cross-disciplinary skills section of the
Texas College and Career Readiness Standards compare to what is currently
taught in a range of entry-level Career and Technical Education course areas at
Texas institutions of higher education?
This question was addressed by replicating the research design from the TCCRI Phase
II Validity Study analyzing the relationship between the CCRS and entry-level general
education courses. The validation methodology included: working with institutional
liaisons to nominate CTE instructors to participate in the study, developing an online
document collection and self-ratings tool, collecting course documents (including syllabi,
assignments, assessments, and scoring rubrics) and self-ratings of the level of
necessity of each cross-disciplinary standard for preparation for the course, providing
quality control and technical assistance, and using the results to analyze and report on
the level of alignment between the cross-disciplinary standards and the CTE courses.
The key differences between the TCCRI Phase II Validity Study and this study are
twofold. First, the TCCRI Phase II Validity Study examined entry-level general education
courses in English/Language Arts (ELA), mathematics, science, and social studies
around which the content standards of the CCRS were organized. This study examines
a representative range of CTE courses comprising subject matter that varies
significantly from the content standards of the CCRS.
Second, the TCCRI Phase II Validity Study examined the relationship between specific
subject area standards and the related entry-level courses (for example, the ELA
standards were analyzed against entry-level composition and literature courses). That
Improving Alignment Between Postsecondary and Secondary Education
37
report also compared all courses in the core subject areas to the cross-disciplinary
standards. The CTE study compared representative courses only to the crossdisciplinary standards. The content standards included in the CCRS were designed to
represent the knowledge and skills necessary for success in the related entry-level
courses, not the universe of entry-level courses. The cross-disciplinary skills, on the
other hand, were those standards that were found to transcend subject matter, and are
therefore, can be used as a framework for examining all CTE courses.
Data Collection and Analysis Methods
Between November 2008 and January 2009, 136 entry-level CTE course instructors at
43 different postsecondary institutions throughout Texas representing 157 courses in
seven subject areas and nine separate courses rated the importance of each crossdisciplinary standard in relation to their course(s). Instructors teaching entry-level CTE
college courses submitted data through a web-based application.
The data collection process began with the College Readiness Special Advisors at 51
two-year public postsecondary institutions soliciting course nominations from their
respective institutions. The THECB identified course titles that enrolled significant
numbers of students statewide among entry-level CTE students. The THECB selected
courses according to the Texas Common Course Numbering System, a uniform set of
course designations that the majority of Texas institutions of higher education employ to
help facilitate the transfer of entry-level courses between institutions. The course titles
selected consist of the following:
•
•
1
ACNT 1303 Introduction to Accounting I
BMGT 1303 Principles of Management 1
Includes courses with course title MGNT 1303 Principles of Management
Improving Alignment Between Postsecondary and Secondary Education
38
•
•
•
•
•
•
•
DFTG 1309 Basic CAD
DFTG 1405 Technical Drafting
ITSC 1301 Introduction to Computers
ITSC 1401 Introduction to Computers
MRKG 1311 Principles of Marketing
POFI 1301 Computer Applications I
POFT 1301 Business English
Course nominations were collected from the Special Advisors between October 2008
and mid-January 2009. The Special Advisors nominated 211 courses by submitting the
faculty memberʼs name and contact information and the institution-specific course title
when it was known.
In December 2008, instructors whose courses had been nominated received an email
asking them to log in to the online course-submission site. The online course
submission process included the following five steps:
1. Consent to Participate: Participating instructors authorized the use of their
submitted course materials in the creation of a composite Reference Course
Profile. In addition, instructors granted the THECB permission to publish, in part
or in whole, any of the documents that were subsequently incorporated into a
Reference Course Profile. (See Appendix A to view a copy of the consent form.)
2. Course Profile: Participating instructors provided general course information
including course objectives, class size, grading policy, texts used, prerequisite
and pathways courses, and percentage of students who enter their course well
prepared.
3. Course Ratings: Instructors were asked to rate the first three levels of the crossdisciplinary standards. The three top levels were included so as to be able to
analyze the level of necessity of the cross-disciplinary knowledge and skills. For
example, was an entire organizing component (the second level) not necessary
for a CTE course, or just individual performance expectations within the
organizing component? The fourth level of the CCRS, the performance
indicators, are not standards per se, but examples of how the standards could be
demonstrated and measured. As the intent of the performance indicators was for
example purposes only, they were not included in the ratings analysis.
Participating instructors completed an online rating form that asked them to
answer the following question for each cross-disciplinary standard: “How
necessary is this element in preparing students to succeed in my course?”
Improving Alignment Between Postsecondary and Secondary Education
39
Respondents chose one of five options: most necessary, more necessary, less
necessary, least necessary, or not necessary. After selecting a response option
for each standard, instructors then selected one or more rationale statements to
explain the reason they rated the item the way that they did. (See Appendix B for
a list of scale items and rationale statements.) The rationale statements were
included to provide greater clarity of understanding of responses. For example,
an instructor might designate a standard as not necessary or least necessary for
one of several reasons. For example, the standard might not be necessary to
succeed in the course because it was irrelevant to the subject area, or it might be
covered in a subsequent course. The rationale statements were particularly
valuable in interpreting the reasons why specific standards were found to be not
well aligned.
4. Additional Questions: Participating instructors also responded to a set of specific
questions to collect data on common components of and current practices in
entry-level Career and Technical Education courses. (See Appendix C for the list
of additional questions.)
5. Upload Course Materials: Participating instructors uploaded key course
documents, including syllabi, assignments, assessments, grading rubrics and
any other relevant materials. All identifying information was removed.
Overall, instructors at 43 separate public two-year postsecondary institutions throughout
the state of Texas completed course submissions. Table 5 presents an overview of the
disposition of all nominated courses.
Table 5: Final Course Status for All Nominated Courses
Completed
Course
Nomination
Partial
Completed
Course
Submission
Declined
Participation
No
Response
Total
ACNT 1303 Introduction to
Accounting I
25
0
1
9
35
BMGT 1303 Principles of
Management
20
0
0
11
31
DFTG 1309 Basic CAD
20
1
0
3
24
DFTG 1405 Technical Drafting
15
0
0
5
20
ITSC 1301 Introduction to
Computers
11
1
0
8
20
ITSC 1401 Introduction to
Computers
9
1
0
3
13
MRKG 1311 Principles of
Marketing
19
0
2
6
27
Course Title
Improving Alignment Between Postsecondary and Secondary Education
40
Course Title
Completed
Course
Nomination
Partial
Completed
Course
Submission
Declined
Participation
No
Response
Total
POFI 1301 Computer Applications I
17
0
0
1
18
POFT 1301 Business English
21
0
0
2
23
Total
157
3
3
48
211
Courses submitted from an individual institution ranged from 1 to 14. The average
number of courses received from participating campuses was 3.7. Table 3 summarizes
the distribution of course submissions by institution type and region.
Table 6: Distribution of all Course Submissions by Region and Institution Type
Region
Community College
Technical College
Total
Central
27
0
27
Gulf Coast
20
0
20
High Plains
5
0
5
Metroplex
35
0
35
Northwest
5
0
5
South
12
4
16
Southeast
7
8
15
Upper East
13
2
15
Upper Rio Grande
4
0
4
West
15
0
15
Total
143
14
157
Ratings
To determine the level of alignment, the modal (most frequent) instructor response was
determined for each individual cross-disciplinary standard. This approach is consistent
with the methodology employed in the TCCRI Phase II study. Each standard could have
a score of 1 through 5 (from most necessary to not necessary). Depending upon the
level of necessity selected, instructors then selected rationale statements that best
explained their responses. (See Appendix D for a list of scale items and rationale
statements.) Items left blank by instructors were treated as missing data. The overall
Improving Alignment Between Postsecondary and Secondary Education
41
results of the faculty ratings are presented in the next section. For a detailed course-bycourse breakdown of the results, please refer to the original Phase I CTE Study.
Results
Overall, the findings from this study indicate that every CCRS cross-disciplinary
standard is aligned with at least one of the nine CTE courses analyzed. The level of
alignment (including standards deemed either necessary for preparation or covered in
the course) between the full set of cross-disciplinary standards, and the nine CTE
course titles analyzed ranges from 100 percent in DFTG 1405 Technical Drafting to 66
percent in POFT 1301 Business English. While the level of alignment of the crossdisciplinary CCRS and any single course included varies, an examination across all
CTE courses studied reveals high alignment between the cross-disciplinary skills across
a range of typical entry-level CTE coursework. The overall alignment results are
presented in Table 7.
Table 7: Summary of Alignment of Cross-disciplinary CCRS Deemed Necessary or
Taught in the CTE Courses2
Course Title
2
Percent of Cross-Disciplinary Standards
Necessary or Taught
ACNT 1303 Introduction to Accounting I
81%
BMGT 1303 Principles of Management
84%
DFTG 1309 Basic CAD
84%
DFTG 1405 Technical Drafting
100%
ITSC 1301 Introduction to Computers
97%
ITSC 1401 Introduction to Computers
84%
MRKG 1311 Principles of Marketing
98%
POFI 1301 Computer Applications I
76%
POFT 1301 Business English
66%
May not total to 100 percent due to rounding error.
Improving Alignment Between Postsecondary and Secondary Education
42
To determine the overall level of alignment, the modal ratings and rationale responses
were combined. When looking at the specific rationales behind the instructor ratings, a
pattern emerges. The typical (modal) rationale for all of the inconsistently aligned
standards were described as being less necessary for successful preparation because
the element was expected to be taught in the course. All of these inconsistently aligned
standards were considered appropriate content. Since all of the inconsistently aligned
standards were considered appropriate content for the courses, all aligned and
inconsistently aligned standards were included in the overall results.
Highly Aligned Organizing Components and Performance Expectations
Of the 11 organizing components of the cross-disciplinary CCRS, five are highly aligned
to CTE courses. Highly aligned is determined by alignment in each of the nine CTE
courses included in this study. Table 8 below lists the five cross-disciplinary organizing
components highly aligned across all CTE courses.
Table 8: Organizing Components Highly Aligned Across All CTE Courses by Rank
Rank
Cross-Disciplinary
Standard
Total
Responses
Total
"Most"
Responses
Total "More"
Responses
Total
Aligned
Responses
("Most" or
"More")
Percent
Aligned
Responses
1.
I.E. Work habits
138
72
59
131
95%
2.
I.D. Academic
behaviors
137
56
67
123
90%
3.
I.F. Academic integrity
139
69
49
118
85%
4.
II.E. Technology
152
63
54
117
77%
5.
II.A. Reading across
the curriculum
150
37
75
112
75%
Of the 45 performance expectations of the cross-disciplinary CCRS, 12 are highly
aligned across the nine CTE courses. Table 9 below lists, in order of the strength of
alignment, the highly aligned performance expectations. These aligned standards are
split equally between Key Cognitive Skills and Foundational Skills, suggesting that each
of these areas is equally valuable for preparation for success in CTE courses.
Improving Alignment Between Postsecondary and Secondary Education
43
Table 9: Performance Expectations Highly Aligned Across All CTE Courses by Rank
Rank
Cross-Disciplinary Standard
Total
Responses
Total
"Most"
Responses
Total
"More"
Responses
Total
Aligned
Responses
("Most" or
"More")
Percent
Aligned
Responses
1.
I.E.1. Work independently.
145
75
59
134
92%
2.
I.D.2. Use study habits
necessary to manage
academic pursuits and
requirements.
144
63
66
129
90%
3.
I.D.4. Persevere to complete
and master tasks.
144
80
49
129
90%
4.
II.A.4. Identify the key
information and supporting
details.
153
54
82
136
89%
5.
I.F.4. Understand and adhere
to ethical codes of conduct.
145
83
44
127
88%
6.
I.D.1. Self-monitor learning
needs and seek assistance
when needed.
143
57
68
125
87%
7.
II.E.4. Use technology
appropriately.
153
66
56
122
80%
8.
II.A.2. Use a variety of
strategies to understand the
meanings of new words.
153
39
81
120
78%
9.
II.A.1. Use effective
prereading strategies.
153
32
79
111
73%
10.
II.E.3. Use technology to
communicate and display
findings in a clear and
coherent manner.
153
56
54
110
72%
11.
II.E.1. Use technology to
gather information.
153
47
57
104
68%
12.
I.F.1. Attribute ideas and
information to source materials
and people.
144
45
48
93
65%
Improving Alignment Between Postsecondary and Secondary Education
44
Significance
The findings of this study do indicate that it is reasonable to assume the crossdisciplinary skills are necessary for career readiness. It is predictable that some of the
individual courses examined did have lower levels of alignment (such as POFT 1301
Business English). Individual courses are often narrow in scope and not representative
of the full constellation of knowledge and skills developed in a full technical program
associated with a given career path. This study tested the assumption that Key
Cognitive and Foundational Skills are necessary for CTE preparation, and the data
strongly suggest that students do need to possess a full array of cognitive and
behavioral skills and strategies to be successful in a range of CTE courses.
The snapshot of CTE courses examined provides an informative picture of the
knowledge and skills currently necessary for preparation in a range of entry-level CTE
courses. For example, five of the CCRS cross-disciplinary organizing components were
found to be highly aligned across all the CTE courses included in this study. These five
organizing components include (in rank of order of alignment level): work habits,
academic behaviors, academic integrity, technology, and reading across the curriculum.
This sort of clear finding can help inform high school faculty immediately regarding the
kinds of skills that are important to emphasize in courses containing students who seem
likely to go on to CTE programs upon completion of high school. The identified
organizing components could be emphasized throughout the four years of high school in
all classes, with special attention to their implementation in high school CTE courses.
For postsecondary instructors and administrators, this study offers insight into current
expectations and practice in a range of CTE courses. The results of this study are
augmented by the CTE Reference Course Profiles that were also generated based upon
the results of this study and a range of additional course documents collected. The
results of this study, along with the Profiles, can potentially serve as a reference point
for postsecondary instructors and course planners who have responsibility for entry-
Improving Alignment Between Postsecondary and Secondary Education
45
level CTE courses. While instructors retain academic freedom for the way they teach
their courses, the CTE Reference Course Profiles can provide an additional point of
comparison instructors can use when creating or refining entry-level courses. The goal
is to assist all CTE postsecondary programs to ensure that the entry-level courses they
offer are aligned with the CCRS, contain appropriate content, and are cognitively
challenging.
Finally, the study also suggests a potential gap between the current level of preparation
of entering CTE students and an optimal level of preparation for future students. If the
goal is to have more students attend and succeed in CTE programs, then proficiency
with the Key Cognitive and Foundational Skills addressed by the cross-disciplinary
standards could fundamentally enhance their success and enable CTE instructors to
teach more efficiently and effectively. The end result would be courses in which
students learn more and in which more students succeed.
Improving Alignment Between Postsecondary and Secondary Education
46
Project #5:
Alignment Between the CCRS and Placement Tests
Commonly Used in Texas Postsecondary Institutions
Overview
Although the number of high school graduates pursuing higher education has increased
during the last 30 years, between 30 percent and 60 percent of college freshmen
require remedial education (NCES, 2004). A student may be college-eligible—that is,
able to meet college admissions requirements—without being college ready (Conley,
2005). To be college ready, a student must attain a level of preparation to enroll and
succeed—without remediation—in credit-bearing general education courses at the
postsecondary level (Conley, 2007). Not only does remedial education cost an
estimated $1 billion annually, but students requiring it are far less likely to graduate
(ACT, 2005). As a result, significant attention is now directed to understanding and
remedying the gap between what high school graduates know and are capable of and
what entry-level college students need to know and are capable of.
Addressing the lack of preparedness of high school graduates for college-level
coursework requires identifying the gaps in content between what is taught in high
school and what is expected of students in college. It also requires appropriate and
comprehensive measures for educators to monitor progress toward closing this gap.
One option for monitoring the developing college preparedness of high school students
is through state assessments. Required state high school assessments often shape
what gets emphasized in classrooms. Many states do not yet explicitly align the content
knowledge and skills assessed by the state tests with those required for college
success. Conley (2003) and Brown and Conley (2007) evaluated state tests and found
that the content of state high school tests is inconsistently and only moderately aligned
with college readiness standards in 20 states in mathematics and ELA. They find
slightly more alignment with the ELA exams than with the mathematics exams, yet the
Improving Alignment Between Postsecondary and Secondary Education
47
ELA exams align poorly or not at all in areas requiring higher levels of thinking, and the
mathematics exams have the highest alignment in basic skills content areas. The
findings describe the gap between student learning expectations and what they are
expected to know to be ready for college and provide evidence that state tests are not
aligned well enough with college readiness standards to be useful for providing
feedback to high school students and teachers concerning college readiness.
Consequently, many students graduating from high school and passing their stateʼs
high-stakes assessments are not sure how well prepared they are for postsecondary
coursework.
College admissions and placement tests, then, become more important means for
assessing college preparedness. Colleges and universities use scores on placement
exams for a variety of purposes but most often for determining whether a student should
have to take a remedial course before being allowed access to credit-bearing, collegelevel courses. Even though these exams are explicitly college-focused, evidence
suggests that they are only moderately aligned with college readiness standards. Brown
and Niemi (2007), for example, investigated the alignment of high school test content
and community college test placement content (ACCUPLACER and COMPASS) and
find that the content of ELA tests are adequately aligned but that mathematics tests are
aligned only in depth-of-knowledge consistency. This study focuses on community
college expectations, however, and the results may not generalize fully to four-year
institutions. Achieve, Inc. (2007), also analyzes the alignment of admissions and
placement exams in mathematics and English (ACT, SAT, ACCUPLACER, COMPASS,
THEA) with the American Diploma Project (ADP) benchmarks. Their analysis, which
may be considered more generalizable, finds that the ACT and SAT are generally more
demanding than placement tests and demonstrate better coverage of a range of content
knowledge. All the tests fell short of measuring the full set of knowledge and skills
encompassed by the ADP benchmarks. Overall, the study finds that the tests place too
much emphasis on low-level content taught earlier in a studentʼs high school education
Improving Alignment Between Postsecondary and Secondary Education
48
or even in middle school.
Texas is a leader in the movement to align high school curricula and assessments with
college readiness expectations and was one of the first states to adopt a specific and
comprehensive definition of college readiness describing essential knowledge and skill
standards for all students in the state.
With the foundation of college and career readiness standards firmly in place, Texas is
prepared to take the next steps in improving college readiness: to ascertain the
alignment of postsecondary placement with the CCRS. By doing so, the THECB can
ascertain the alignment of its measures of placement with the state standards and can
modify placement procedures and practices so that incoming students who focus on
meeting the CCRS while in high school are placed into college courses accordingly.
The purpose of the Texas Test Alignment Project (TAP) is to examine the alignment of
THECB-approved college admission and placement tests used by Texas institutions of
higher education to measure college readiness with the CCRS. As such, the goal of the
current study is not to identify the best test to assess the CCRS, but to ascertain the
discrepancy between current practice and the desired goal of complete measurement of
the CCRS. This study addresses the following research objectives:
•
To what extent do college admissions and placement tests assess the
knowledge and skills specified in college and career readiness standards?
•
Are any areas of college and career readiness standards not assessed by
placement and admissions tests?
•
How do the tests compare to college and career ready standards in terms of rigor
and cognitive demand?
Methodology
Test items from the six THECB-approved placement and admissions tests used by
Texas institutions of higher education to measure college readiness were obtained for
the study. Content experts in mathematics and ELA first rated the rigor and depth of
Improving Alignment Between Postsecondary and Secondary Education
49
knowledge required by the CCRS. The CCRS are organized into three levels: key
content areas, organizing components, and performance expectations. The reviewers
made their judgments at the level of performance expectations by rating the cognitive
complexity of each performance expectation and then determining whether each test
item from each of the designated tests corresponds with one or more performance
expectations. Every reviewer determined the relationship between each test item and
the corresponding performance expectations. The results show the average cognitive
complexity ratings and describe the relationship between performance expectations and
test itemʼs content (their categorical concurrence) and, when they match, whether rigor
and depth of knowledge between the performance expectation and the test item is
appropriate.
Researchers have developed many ways to assess alignment (Rothman, 2004; Webb,
1997, 1999; Porter, 2002; ACHIEVE, 2000, 2001, and 2003; Wixson, Fisk, Dutro, and
McDaniel, 2002). Webbʼs method is one of the more widely used ways to determine
alignment between test items and educational standards. His approach however, is
often modified by alignment researchers (Impara, 2001).
This study also is based on Webbʼs methods, but were modified to better address the
projectʼs objectives. The primary deviation is to compute each metric across raters
rather than by computing alignment metrics for each rater and then averaging them.
Although the estimates may be slightly less sensitive to differences in reviewer ratings,
they highlight the alignment between each of the tests and the CCRS—as this
alignment is a core objective of the study.
•
Categorical concurrence determines the extent that test items correspond to
the CCRS. Tests with more than six items assessing a standard are considered
to demonstrate categorical concurrence with that standard.
•
Depth-of-knowledge (DOK) consistency is measured by comparing the DOK
level required of the test item (as determined by the reviewers) with the DOK
required to meet the standard (as determined by reviewers). The recommended
benchmark is that at least one-half the assessment items are at or above the
Improving Alignment Between Postsecondary and Secondary Education
50
DOK required by the corresponding standard.
•
Range of knowledge is the extent to which the CCRS are assessed by items on
a test. This criterion is typically measured as the average percent of standards
addressed by one or more items on each test (across reviewers).
Webb includes a fourth measure, balance of representation, to describe the extent that
assessment items are evenly distributed across learning objectives. Previous research
provides evidence that this is highly redundant with the range of knowledge measure
(e.g., Brown and Conley, 2007). Furthermore, when there is great variation in test
distribution (e.g., the number of items provided to represent each test), as in the current
study, this distribution metric is more likely to be skewed and not be directly comparable
across tests. For these reasons, balance of representation is not reported here.
Results
Reviewer Agreement
As is typical with alignment studies, the ratings of six experts were used as the basis for
alignment computations and decisions. Six is the standard number of reviewers required
in similar alignment studies to obtain sufficient reliability (Herman, Webb and Zuniga,
2005; Webb 1997, 1999, and 2002). Consistent with the treatment of rigor ratings and
cognitive demand ratings as quasi-quantitative measures (rather than as strictly
categorical measures), a generalizability analysis was conducted with items and
standards crossed by reviewers (Shavelson and Webb, 1991).
The absolute error variance indicates the consistency of item-and-standard ratings
across reviewers. The phi coefficient, called the index of dependability (Shavelson and
Webb, 1991), can be considered a reliability-like coefficient for absolute decisions
(Herman, Webb and Zuniga, 2007; Mushquash and OʼConnor, 2006; Thompson, 2003).
The phi coefficient in this context represents decision consistency over parallel rating
situations. For example, a phi coefficient of 0.80 indicates that, if the same study was
recreated and repeated over multiple occasions, one could expect the same pattern of
Improving Alignment Between Postsecondary and Secondary Education
51
ratings 80% of the time. Coefficients are interpreted similarly to reliability coefficients in
that values of 0.80 and higher indicate high generalizability.
Table 10 shows the generalizability results for the rigor and cognitive-demand ratings of
the mathematics and English items across assessments. The phi coefficients for the
cognitive-demand ratings are close to or above the conventional 0.80 criterion for
reliability. The phi coefficients for the rigor ratings were lower than those for the
cognitive-demand ratings, with none of the coefficients reaching the conventional
criterion for reliability. The six reviewers obtained an acceptable level of dependability
for estimating mathematics and ELA itemsʼ level of cognitive demand, but were much
less consistent in their rating of the level of rigor.
Table 10. G-Study Coefficients for Mathematics and ELA Items across Test
Subject
Items
Mathematics
ELA
Cognitive Demand
Rigor
Phi Coefficients
Absolute Error Variance
Phi Coefficients
Absolute Error Variance
1,460
0.859
0.035
0.505
0.007
1,239
0.703
0.053
0.446
0.02
Table 11 shows the generalizability results for the cognitive demand ratings and rigor
ratings of the mathematics and ELA CCRS performance expectations. As with the item
ratings, the phi coefficients for the performance expectation cognitive-demand ratings
are close to or above the conventional 0.80 criterion for reliability. The phi coefficients
for the rigor ratings were lower than those for the cognitive demand ratings, with none of
the coefficients reaching the conventional criterion for fully satisfactory reliability. These
results indicate that the six reviewers did reach an acceptable level of dependability for
estimating mathematics and ELA standardsʼ level of cognitive demand but not their level
of rigor.
Improving Alignment Between Postsecondary and Secondary Education
52
Table 11. G-Study Coefficients for Mathematics and ELA CCRS Performance
Expectations Ratings
Cognitive Demand
Rigor
Subject
Standards
Phi
Coefficients
Absolute Error
Variance
Phi
Coefficients
Absolute Error
Variance
Mathematics
115
0.855
0.100
0.566
0.038
ELA
119
0.724
0.095
0.556
0.060
Overall, these results indicate higher rater reliability for mathematics item ratings and
standard ratings than for ELA item ratings and standard ratings. Additionally, the results
indicate higher rater reliability for cognitive-demand ratings than for rigor ratings. Results
are consistent with findings from other similar studies using the same number of
reviewers (Herman, Webb, and Zuniga, 2005, Brown and Conley, 2007, Brown and
Niemi, 2007). Interestingly, Herman, Webb, and Zuniga also reported that ratings of
cognitive demand were more reliable than were ratings of centrality (similar to rigor in
that centrality evaluated the extent that a standard was essential to a topic).
A D-study (decision study) was also conducted, which determined the gains in reliability
expected if the number of reviewers was increased in future studies. The results
indicate that, in most cases, increasing the number of reviewers from 6 to 10–15 would
result in only moderate gains in reliability (as measured by the phi coefficient).
Findings
Study findings are described below beginning with a summary of the rigor and cognitive
demand of the CCRS and the tests, including:
•
Average rigor and cognitive demand of CCRS organizing components
•
Average rigor and cognitive demand of tests, across items
Also described is the alignment between the test items and the CCRS, specifically:
•
The total number of matches or hits for each performance expectation
•
Zero-match performance expectations, or those not assessed by any of the tests
Improving Alignment Between Postsecondary and Secondary Education
53
•
Categorical concurrence, modified to describe when the number of items
assessing a performance expectation on each test is six or more for mathematics
and ELA tests
•
Depth-of-knowledge consistency for mathematics and ELA tests, describing
average differences in rigor and cognitive-demand ratings
•
Range of knowledge for mathematics and ELA tests identifying the total number
of matches for each performance expectation by test
Rigor and Cognitive Demand
Texas Career and College Readiness Standards Rigor and Cognitive Demand
Reviewers provided rigor and cognitive-demand ratings for each of the CCRS
performance expectations. Each reviewerʼs ratings were averaged across the expanded
performance expectations and then averaged across reviewers to generate rigor and
cognitive-demand scores for each item and then for each performance expectation.
Table 12 provides the average of reviewersʼ rigor and cognitive-demand ratings across
performance expectation for each organizing component of the CCRS cross-disciplinary
skills. On average, the reviewers found the cross-disciplinary skills to be at or below the
level at which an entry-level college student should perform (between 1 and 2 on the
rigor scale). Reviewers rate the key cognitive strategies at a higher level of rigor and
cognitive demand than the foundational skills. Note the similar patterns in findings
between rigor and cognitive demand; however, given the higher reliability of the
cognitive demand ratings, somewhat greater consideration should be given to cognitive
demand ratings than to rigor ratings.
Table 12. Average Rigor and Cognitive Demand for Cross-Disciplinary Skills by
Organizing Component
Organizing Component
Average Rigor
(3-point scale)
Average Cognitive Demand
(4-point scale)
Mean
N
SD
Mean
N
SD
Intellectual curiosity
2.04
2
0.177
3.29
2
0.059
Reasoning
1.92
4
0.180
3.35
4
0.299
Key Cognitive Strategies
Improving Alignment Between Postsecondary and Secondary Education
54
Organizing Component
Average Rigor
(3-point scale)
Average Cognitive Demand
(4-point scale)
Mean
N
SD
Mean
N
SD
Problem solving
1.89
3
0.192
3.33
3
0.220
Academic behaviors
1.40
4
0.125
2.38
4
0.285
Work habits
1.13
2
0.059
2.46
2
0.059
Academic integrity
1.60
4
0.197
2.58
4
0.642
Overall Key Cognitive Strategies Mean
1.67
19
0.329
2.88
19
0.549
Reading across the curriculum
1.23
8
0.165
2.05
8
0.478
Writing across the curriculum
1.14
3
0.127
2.00
3
0.500
Research across the curriculum
1.59
8
0.225
2.59
8
0.384
Use of data
1.81
3
0.268
2.94
3
0.459
Technology
1.29
4
0.144
1.96
4
0.567
Overall Foundational Skills Mean
1.41
26
0.287
2.30
26
0.554
Foundational Skills
Note: Averages were computed for each performance expectation across reviewers and then were averaged across
performance expectations within each organizing component. “N” refers to the number of performance expectations
within each organizing component. The standard deviations (SD) describe the variability of the performance
expectations within organizing components. Shaded cells indicate the organizing components with the highest
average rigor or cognitive demand ratings.
Within the cross-disciplinary skills, the key cognitive strategies were on average, more
rigorous and cognitively demanding than the foundational skills.
Table 13 provides the average of reviewersʼ rigor and cognitive-demand ratings across
performance expectations for each of the mathematics organizing components. The
rigor ratings are slightly lower for mathematics than for the cross-disciplinary skills. On
average, the reviewers rated the mathematics standards as below the level at which an
entry-level college student should be expected to perform (near 1 on the rigor scale).
Table 13. Average Rigor and Cognitive Demand for Mathematics by Organizing
Component
Organizing Component
Average Rigor
(3-point scale)
Average Cognitive Demand
(4-point scale)
Mean
N
SD
Mean
N
SD
Number representation
1.08
2
0.118
1.00
2
0.000
Number operations
1.17
1
1.00
1
Number sense and number concepts
1.00
1
1.83
1
Numeric Reasoning
Improving Alignment Between Postsecondary and Secondary Education
55
Organizing Component
Average Rigor
(3-point scale)
Average Cognitive Demand
(4-point scale)
Mean
N
SD
Mean
N
SD
1.08
4
0.096
1.21
4
0.417
Expressions and equations
1.17
1
1.67
1
Manipulating expression
1.00
1
1.17
1
Solving equations, inequalities, and systems of
equations
1.08
2
0.118
1.58
2
0.589
Representations
1.08
2
0.118
2.00
2
0.000
Overall Algebraic Reasoning
1.08
6
0.091
1.67
6
0.408
Figures and their properties
1.11
3
0.192
1.78
3
1.072
Transformations and symmetry
1.11
3
0.192
1.61
3
0.770
Connections between geometry and other
mathematical content strands
1.17
3
0.167
1.83
3
0.167
Logic and reasoning in geometry
1.17
2
0.000
2.42
2
0.825
Overall Geometric Reasoning
1.14
11
0.146
1.86
11
0.710
Measurement involving physical and natural
attributes
1.00
1
1.17
1
Systems of measurement
1.00
2
0.000
1.00
2
0.000
Measurement involving geometry and algebra
1.06
3
0.096
1.22
3
0.385
Measurement involving statistics and probability
1.25
2
0.118
1.92
2
1.296
Overall Measurement Reasoning
1.08
8
0.126
1.33
8
0.649
Counting principles
1.17
1
1.17
1
Computation and interpretation of probabilities
1.33
2
0.236
1.50
2
0.471
Overall Probabilistic Reasoning
1.28
3
0.192
1.39
3
0.385
Data collection
1.67
1
2.50
1
Describe data
1.04
4
0.083
1.54
4
0.479
Read, analyze, interpret, and draw conclusions
from data
1.50
4
0.136
2.67
4
0.304
Overall Statistical Reasoning
1.31
9
0.282
2.15
9
0.674
Recognition and representation of functions
1.00
2
0.000
1.17
2
0.236
Analysis of functions
1.08
2
0.118
2.25
2
0.589
Model real world situations with functions
1.33
2
0.471
2.33
2
0.707
Overall Functions
1.14
6
0.267
1.92
6
0.721
Overall Numeric Reasoning
Algebraic Reasoning
Geometric Reasoning
Measurement Reasoning
Probabilistic Reasoning
Statistical Reasoning
Functions
Improving Alignment Between Postsecondary and Secondary Education
56
Organizing Component
Average Rigor
(3-point scale)
Average Cognitive Demand
(4-point scale)
Mean
N
SD
Mean
N
SD
Mathematical problem solving
1.30
5
0.139
3.10
5
0.418
Logical reasoning
1.50
2
0.236
3.50
2
0.471
Real world problem solving
1.33
3
0.167
2.83
3
0.601
Overall Problem Solving and Reasoning
1.35
10
0.166
3.10
10
0.492
Connections among the strands of mathematics
1.25
2
0.118
2.67
2
0.000
Connections of mathematics to nature, real-world
situations, and everyday life
1.50
3
0.167
2.50
3
0.167
Overall Connections
1.40
5
0.190
2.57
5
0.149
Language, terms, and symbols of mathematics
1.06
3
0.096
2.22
3
0.674
Interpretation of mathematical work
1.25
2
0.118
2.25
2
0.354
Presentation and representation of mathematical
work
1.33
3
0.167
2.61
3
0.509
Overall Communication and Representation
1.21
8
0.173
2.38
8
0.510
Problem Solving and Reasoning
Connections
Communication and Representation
Note: Averages were computed for each performance expectation across reviewers and then were averaged across
performance expectations within each organizing component. “N” refers to the number of performance expectations
within each organizing component. The standard deviations (SD) describe the variability of the performance
expectations within organizing components. Shaded cells indicate the organizing components with the highest
average rigor or cognitive demand ratings.
The organizing components with the highest average cognitive demand ratings belong
to the statistical reasoning, problem solving and reasoning, and statistical reasoning key
content areas, including the connections and problem solving and reasoning, data
collection, read, analyze, interpret, and draw conclusions from data, logical reasoning,
and connections of mathematics to nature, real-world situations, and everyday life
organizing components. The organizing components with the highest cognitive demand
ratings tend also to be high in rigor.
Table 14 provides the average of reviewersʼ rigor and cognitive-demand ratings across
performance expectations for each of the ELA organizing components. On average, the
ELA standards are at the level an entry-level college student should perform (close to 2
on the rigor scale).
Improving Alignment Between Postsecondary and Secondary Education
57
Table 14. Average Rigor and Cognitive Demand for English/Language Arts by Organizing
Component
Average
Rigor
Organizing Component
Average
Cognitive Demand
Mean
N
SD
Mean
N
SD
Compose a variety of texts that demonstrate clear focus, the
logical development of ideas in well-organized paragraphs, and
the use of appropriate language that advances the authorʼs
purpose
1.83
5
0.199
2.42
5
0.391
Overall Writing
1.83
5
0.199
2.42
5
0.391
Locate explicit textual information and draw complex
inferences, analyze, and evaluate the information within and
across texts of varying lengths
1.59
11
0.277
2.35
11
0.490
Understand new vocabulary and concepts and use them
accurately in reading, speaking, and writing
1.22
3
0.096
1.50
3
0.333
Describe, analyze, and evaluate information within and across
literary and other texts from a variety of cultures and historical
periods
1.83
4
0.381
2.51
4
0.790
Explain how literary and other texts evoke personal experience
and reveal character in particular historical circumstances
1.92
2
0.589
2.75
2
0.825
Overall Reading
1.62
20
0.355
2.30
20
0.637
Formulate topic and questions
1.53
3
0.127
2.47
3
0.173
Select information from a variety of sources
1.81
4
0.336
2.60
4
0.448
Produce and design a document
1.75
2
0.118
2.71
2
0.059
Overall Research
1.70
9
0.257
2.58
9
0.303
Writing
Reading
Research
Note: Averages were computed for each performance expectation across reviewers and then were averaged across
performance expectations within each organizing component. Standard deviations describe the variability of the
performance expectations within organizing components. Shaded cells indicate the organizing components with the
highest average rigor or cognitive demand ratings.
For ELA, writing and research are rated as more rigorous and cognitively demanding
than reading, and the average rigor and cognitive demand for the ELA organizing
components are higher than for the mathematics and the cross-disciplinary skill
organizing components.
Test Rigor and Cognitive Demand
The reviewers found tests overall to be nearly identical in their level of rigor, which on
average was below the level of the CCRS and below the level at which an entry-level
Improving Alignment Between Postsecondary and Secondary Education
58
college student should perform. This should not come as a surprise because
admissions and placement tests must cover a range of knowledge and skill levels to
uncover test takers who are below the level of a credit-bearing college course. To do so
requires at least some items that are below college level. This explains why, in most
cases, the average test items were less rigorous than the standards.
Table 15 describes the overall average rigor ratings for each of the tests by subject.
Table 15. Average Rigor for Mathematics and English/Language Arts by Test
Test
Average Mathematics Item Rigor
Average ELA Item Rigor
(Averaged across rater and items)
(Averaged across rater and items)
Mean
N
SD
Mean
N
SD
A
1.08
248
0.139
1.21
162
0.142
B
1.03
180
0.098
1.16
347
0.179
C
1.02
528
0.070
1.08
240
0.110
D
1.06
351
0.146
1.08
240
0.158
E
1.08
100
0.124
1.21
167
0.178
F
1.05
53
0.107
1.26
83
0.190
There is greater variability in the overall cognitive-demand ratings. Table 16 describes
the average cognitive-demand ratings for each test by subject.
Table 16. Average Cognitive Demand for Mathematics and English/Language Arts by Test
Test
Average Mathematics Item Cognitive Demand
Average ELA Item Cognitive Demand
(Averaged across rater and item)
(Averaged across rater and item)
Mean
N
SD
Mean
N
SD
A
1.39
248
0.376
1.64
162
0.352
B
1.84
180
0.487
1.37
347
0.427
C
1.50
528
0.502
1.28
240
0.341
D
1.56
351
0.475
1.42
240
0.389
E
2.05
100
0.369
1.46
167
0.42
F
1.73
53
0.463
1.59
83
0.421
Within the ELA tests, the constructed-response writing prompts received higher rigor
and cognitive demand ratings than did the multiple-choice reading items (with some
variability in ratings by test). These were also more cognitively demanding and were the
Improving Alignment Between Postsecondary and Secondary Education
59
only items to consistently require the highest level of cognitive demand.
Alignment Between Test Items and CCRS
Test Coverage of CCRS
Reviewers identified a total of 57,391 hits or matches between the 2,699 test items and
the 149 CCRS performance expectations. As expected, the test items assessed multiple
standards, ranging from 15 to 34. Across tests, the average number of matches per test
item ranged from 18 to 34 for mathematics and from 15 to 18 for ELA.
Most- and Least-Assessed CCRS Organizing Components
The most hits occurred for the key content areas of writing, problem solving and
reasoning, algebraic reasoning, and foundational skills. The standards with the fewest
test items assessing them are research, probabilistic reasoning, and statistical
reasoning.
Zero-Match Expectations
Expectations not assessed by any test are called zero-match performance expectations.
Table 17 identifies the CCRS items not assessed by a single test item on any of the
tests.
Table 17. Zero-Match CCRS Performance Expectations
Key Content
Mathematics:
Statistical
Reasoning
Organizing Components
Performance Expectation
Not Assessed by a Single Test Item
A. Data collection
1. Plan a study
C. Read, analyze, interpret, and draw
conclusions from data
3. Analyze relationships between paired data using
spreadsheets, graphing calculators, or statistical
software
4. Recognize reliability of statistical results
ELA:
Reading
A. Locate explicit textual information
and draw complex inferences, analyze,
and evaluate the information within and
across texts of varying lengths
B. Understand new vocabulary and
concepts and use them accurately in
reading, speaking, and writing
8. Compare and analyze how generic features are
used across texts
2. Apply knowledge of roots and affixes to infer the
meanings of new words
3. Use reference guides to confirm the meanings of
new words or concepts
Improving Alignment Between Postsecondary and Secondary Education
60
Key Content
Organizing Components
C. Describe, analyze, and evaluate
information within and across literary
and other texts from a variety of cultures
and historical periods
D. Explain how literary and other texts
evoke personal experience and reveal
character in particular historical
circumstances
Performance Expectation
Not Assessed by a Single Test Item
1. Read a wide variety of texts from American,
European, and world literatures
3. Analyze works of literature for what they suggest
about the historical period and cultural contexts in
which they were written
2. Analyze the influence of myths, folktales, fables,
and classical literature from a variety of world
cultures on later literature and film
1. Formulate research questions
A. Formulate topic and questions
3. Refine research topic and devise a timeline for
completing work
ELA:
Research
2. Explore a research topic
B. Select information from a variety of
sources
C. Produce and design a document
4. Use source material ethically
1. Design and present an effective product
2. Use source material ethically
Note: The CCRS include “Use source material ethically” in both the select information from a variety of sources and
produce and design a document organizing components.
Many of these appear to be skills best assessed by means other than multiple-choice or
even constructed-response type test items: plan a study, use reference guides to
confirm the meanings of new words, formulate research questions, explore a research
topic, use source material ethically, etc. These are not included as in subsequent
descriptive tables. However, they are counted when computing measures where the
total number of performance expectations is a denominator, such as with categorical
concurrence.
Categorical Concurrence
Categorical concurrence describes the degree to which the items in a test correspond
with one or more of the CCRS. The criterion of categorical concurrence is met if the
same categories of content appear in both the assessment and the standards; to
produce an acceptable level of reliability for assessment scores, Webb recommends
that at least six items assess each performance expectation. Table 18 describes the
percent of performance expectations demonstrating categorical concurrence across
Improving Alignment Between Postsecondary and Secondary Education
61
rater (i.e., that have at least six items assessing that performance expectation) within
each organizing component for each mathematics test.
Table 18. Categorical Concurrence by Key Content for Mathematics Tests
Key Content
Numeric Reasoning
Algebraic Reasoning
Geometric Reasoning
Measurement
Reasoning
Probabilistic
Reasoning
Statistical Reasoning
Organizing Component
(Number of
performance expectations)
Percent of Performance Expectations
that Reached Categorical Concurrence
for this Test in Mathematics
A
B
C
D
E
F
A. Number representation (2)
50%
50%
100%
50%
50%
0%
B. Number operations (1)
100%
100%
100%
100%
100%
100%
C. Number sense and number
concepts (1)
100%
100%
100%
100%
0%
0%
Numeric Reasoning Overall
83%
83%
100%
83%
50%
33%
A. Expressions and equations (1)
100%
100%
100%
100%
100%
0%
B. Manipulating expression (1)
100%
100%
100%
100%
100%
100%
C. Solving equations, inequalities,
and systems of equations (2)
100%
50%
100%
100%
100%
50%
D. Representations (2)
100%
100%
100%
100%
100%
100%
Algebraic Reasoning Overall
100%
88%
100%
100%
100%
63%
A. Figures and their properties (2)
100%
100%
100%
100%
66%
66%
B. Transformations and symmetry (3)
0%
66%
33%
33%
33%
0%
C. Connections between geometry
and other mathematical content
strands (3)
66%
66%
66%
66%
66%
66%
D. Logic and reasoning in geometry
(2)
0%
100%
100%
100%
0%
0%
Geometric Reasoning Overall
42%
83%
75%
75%
41%
33%
A. Measurement involving physical
and natural attributes (1)
100%
0%
0%
0%
0%
0%
B. Systems of measurement (2)
0%
50%
100%
0%
0%
0%
C. Measurement involving geometry
and algebra (3)
66%
66%
66%
100%
66%
0%
D. Measurement involving statistics
and probability (2)
0%
0%
0%
0%
0%
0%
Measurement Reasoning Overall
42%
29%
42%
25%
17%
0%
A. Counting principles (1)
0%
0%
0%
0%
100%
0%
B. Computation and interpretation of
probabilities (2)
0%
0%
0%
0%
0%
0%
Probabilistic Reasoning Overall
0%
0%
0%
0%
50%
0%
A. Data collection (1)
0%
0%
0%
0%
0%
0%
B. Describe data (4)
0%
25%
25%
25%
100%
0%
Improving Alignment Between Postsecondary and Secondary Education
62
Key Content
Functions
Problem Solving and
Reasoning
Communication and
Representation
Connections
Key Cognitive
Strategies
Foundational Skills
Organizing Component
(Number of
performance expectations)
Percent of Performance Expectations
that Reached Categorical Concurrence
for this Test in Mathematics
A
B
C
D
E
F
C. Read, analyze, interpret, and draw
conclusions from data (4)
0%
0%
0%
0%
25%
0%
Statistical Reasoning Overall
0%
13%
13%
13%
63%
0%
A. Recognition and representation of
functions (2)
50%
0%
50%
50%
0%
0%
B. Analysis of functions (2)
100%
100%
100%
100%
100%
50%
C. Model real world situations with
functions (2)
50%
100%
50%
100%
100%
100%
Functions Overall
67%
67%
67%
83%
67%
50%
A. Mathematical problem solving (5)
80%
80%
80%
80%
80%
80%
B. Logical reasoning (2)
0%
50%
50%
50%
50%
50%
C. Real world problem solving (3)
33%
100%
33%
33%
33%
33%
Problem Solving and
Reasoning Overall
38%
77%
54%
54%
54%
54%
A. Language, terms, and symbols of
mathematics (3)
66%
100%
100%
100%
66%
66%
B. Interpretation of mathematical
work (2)
50%
100%
50%
100%
50%
100%
C. Presentation and representation
of mathematical work (3)
0%
0%
33%
33%
33%
33%
Communication and
Representation Overall
39%
67%
61%
78%
50%
66%
A. Connections among the strands of
mathematics (2)
50%
100%
50%
50%
50%
50%
B. Connections of mathematics to
nature, real-world situations, and
everyday life (3)
33%
66%
33%
33%
33%
33%
Connections Overall
42%
83%
42%
42%
42%
42%
A. Intellectual curiosity (2)
0%
0%
0%
0%
0%
0%
B. Reasoning (4)
0%
0%
0%
0%
0%
0%
C. Problem solving (3)
66%
66%
66%
33%
33%
66%
D. Academic behaviors (4)
0%
0%
0%
0%
0%
0%
E. Work habits (2)
0%
0%
0%
0%
0%
0%
F. Academic Integrity (4)
0%
0%
0%
0%
0%
0%
Key Cognitive Strategies Overall
11%
11%
11%
6%
6%
11%
A. Reading across the curriculum (8)
12%
12%
25%
12%
12%
12%
B. Writing across the curriculum (3)
0%
0%
0%
0%
0%
0%
Improving Alignment Between Postsecondary and Secondary Education
63
Key Content
Overall
Organizing Component
(Number of
performance expectations)
Percent of Performance Expectations
that Reached Categorical Concurrence
for this Test in Mathematics
A
B
C
D
E
F
C. Research across the curriculum
(8)
0%
0%
0%
0%
0%
0%
D. Use of data (3)
0%
0%
0%
0%
0%
0%
E. Technology (4)
0%
0%
0%
0%
0%
0%
Foundational Skills Overall
2%
2%
5%
2%
2%
2%
Average % categorical
concurrence
39%
50%
48%
47%
45%
30%
Note: Average percent categorical concurrence by test computed as the average of the percents across organizing
components. Categorical concurrence is defined as a test having six or more unique test items assessing a
performance expectation and the percents shown are the percent of performance expectations within an organizing
component that attained categorical concurrence for each test. Shaded cells indicate the recommended benchmark
was met.
Reviewers of the mathematics CCRS found categorical concurrence highest across all
tests for algebraic reasoning, functions, and numeric reasoning components and lowest
for the foundational skills. All tests attained categorical concurrence in algebraic
reasoning, functions and communications and representations. Across all tests, the
lowest categorical concurrence was observed in probabilistic reasoning, statistical
reasoning (with one exception), and the cross-disciplinary skills (with the exception of
the problem solving and—to a lesser extent—reading across the curriculum
expectations, which were moderately assessed by the tests).
Although some tests have unique strengths, results are somewhat consistent across
tests: organizing components low in categorical concurrence tend to be low across all
tests, organizing components high in categorical concurrence tend to be high across all
tests. This suggests that, with minor gaps or areas of unique coverage, the placement
and admissions tests provide similar content coverage relative to the CCRS.
Table 19 describes the percent of performance expectations within each organizing
component for which categorical concurrence was established for each
English/language arts test. For the English/language arts CCRS, categorical
concurrence is higher for writing than for the foundational skills and reading.
Improving Alignment Between Postsecondary and Secondary Education
64
It is lowest for research.
Table 19. Categorical Concurrence by Key Content for English/Language Arts Tests
Key Content
Writing
Reading
Research
Key Cognitive
Strategies
Foundational
Skills
Organizing Component
Percent of Performance Expectations
that Reached Categorical Concurrence
for this Test in ELA
A
B
C
D
E
F
I.A. Compose a variety of texts that demonstrate
clear focus, the logical development of ideas in
well-organized paragraphs, and the use of
appropriate language that advances the
authorʼs purpose. (5)
100%
80%
100%
100%
80%
80%
Writing Overall
100%
80%
100%
100%
80%
80%
II.A. Locate explicit textual information and draw
complex inferences, analyze, and evaluate the
information within and across texts of varying
lengths. (11)
55%
73%
46%
46%
46%
46%
II.B. Understand new vocabulary and concepts
and use them accurately in reading, speaking,
and writing. (3)
0%
33%
33%
33%
33%
33%
II.C. Describe, analyze, and evaluate
information within and across literary and other
texts from a variety of cultures and historical
periods. (4)
0%
25%
0%
0%
0%
0%
II.D. Explain how literary and other texts evoke
personal experience and reveal character in
particular historical circumstances.
0%
0%
0%
0%
0%
0%
Reading Overall
14%
33%
20%
20%
20%
20%
V.B. Select information from a variety of
sources. (4)
0%
0%
0%
0%
0%
0%
Research Overall
0%
0%
0%
0%
0%
0%
A. Intellectual curiosity (2)
0%
0%
0%
0%
0%
0%
B. Reasoning (4)
25%
25%
0%
25%
25%
25%
C. Problem solving (3)
33%
33%
33%
33%
33%
33%
D. Academic behaviors (4)
25%
25%
25%
50%
25%
25%
E. Work habits (2)
0%
0%
0%
0%
0%
0%
F. Academic integrity (4)
0%
0%
0%
0%
0%
0%
Key Cognitive Strategies Overall
14%
14%
10%
18%
14%
14%
A. Reading across the curriculum (8)
50%
75%
63%
63%
63%
63%
B. Writing across the curriculum (3)
66%
66%
66%
66%
66%
66%
C. Research across the curriculum (8)
0%
0%
0%
0%
0%
0%
D. Use of data (3)
0%
0%
0%
0%
0%
0%
E. Technology (4)
0%
0%
0%
0%
0%
0%
Improving Alignment Between Postsecondary and Secondary Education
65
Key Content
Overall
Organizing Component
Percent of Performance Expectations
that Reached Categorical Concurrence
for this Test in ELA
A
B
C
D
E
F
Foundational Skills Overall
23%
28%
26%
26%
26%
26%
Average % categorical concurrence
30%
31%
31%
33%
28%
28%
Note: Average percent categorical concurrence by test is computed as the average of the percents across organizing
components. The percents described are the percent of performance expectations within an organizing component
that attained categorical concurrence.
Within the key content areas for reading, one organizing component accounts for nearly
all of the categorical concurrence observed: locate explicit textual information and draw
complex inferences, analyze, and evaluate the information within and across texts of
varying lengths. Within the foundational skills, all tests demonstrate categorical
concurrence on the reading and writing across the curriculum organizing components,
but no tests demonstrate categorical concurrence with any other organizing component
within this key content area.
As with mathematics, the organizing components that are lower in categorical
concurrence trend lower across all tests, and organizing components higher in
categorical concurrence trend higher across all tests, indicating that content coverage is
similar across tests.
Depth-of-Knowledge Consistency
To determine the relationship of rigor and cognitive demand between test-items and the
CCRS, the differences between items and performance-expectation rigor and cognitivedemand rating was computed for all matches. Table 20 summarizes the differences
between the depth of knowledge ratings for test items and the performance
expectations.
Improving Alignment Between Postsecondary and Secondary Education
66
Table 20. Average Differences Between Test-Item Rigor and Cognitive Demand and the
Rigor and Cognitive Demand of the Performance Expectations they Assess
Matches Between Test
Items and Performance
Expectations
Test
Average Difference
in Cognitive Demand
(item cognitive demand
minus performance-expectation
cognitive demand)
Average Difference
in Rigor
(item rigor minus performanceexpectation rigor )
Number
Mean
SD
Mean
SD
A
4,571
-0.38
0.787
-0.10
0.263
B
5,965
-0.18
0.805
-0.17
0.228
C
12,592
-0.25
0.800
-0.16
0.215
D
9,543
-0.36
0.806
-0.12
0.244
E
3,409
-0.16
0.780
-0.13
0.232
F
1,622
-0.44
0.767
-0.15
0.243
A
2,720
-0.54
0.518
-0.33
0.330
B
5,676
-0.73
0.579
-0.31
0.340
C
3,566
-0.82
0.538
-0.40
0.310
D
3,497
-0.64
0.583
-0.38
0.325
E
2,882
-0.67
0.596
-0.28
0.337
F
1,348
-0.56
0.561
-0.23
0.339
Mathematics
English
Note: Average differences are the average of the individual differences between test items and performance
expectations for all matches identified on each test. Negative means indicate that on average, the reviewers rated the
test items lower in cognitive demand or rigor than the performance expectations they assessed. We note that
significance tests are not often used in typical alignment study methodology and that were they to be applied, many
confidence intervals would include zero.
Although all differences are small, the difference between standards and test items in
cognitive demand and rigor is relatively larger for ELA tests than for mathematics tests.
The differences are negative in most cases because the performance expectations are
overall, more rigorous and cognitively demanding than the test items, as is expected
due to the range of rigor required by the tests to assess a range of student proficiency.
The smaller differences indicate closer correspondence between the test items and their
performance expectations.
While on average, the standards were more rigorous than the test items, this was not
the case with all performance expectations. For rigor in mathematics, test items rank
higher than the CCRS in the following categories:
Improving Alignment Between Postsecondary and Secondary Education
67
•
Numeric reasoning, including 1) number representation and 2) number sense
and concepts
•
Algebraic reasoning, including 1) manipulating expressions and 2) solving
equations, inequalities, and systems of equations
•
Measurement reasoning, including 1) measurement involving physical and
natural attributes and 2) systems of measurement
•
Functions, including 1) recognition and representation of functions and 2)
analysis of functions
•
Communication and representation, including language, terms, and symbols of
mathematics
For ELA, the only performance expectation that is rated lower in rigor than assessment
items is writing.
In cognitive demand for both mathematics and ELA, the CCRS are more challenging on
average than are the test items. The exceptions to this, where test items are more
challenging, are:
•
Numeric reasoning, including 1) number representation, 2) number operations
•
Algebraic reasoning, including 1) manipulating expressions, 2) solving equations,
inequalities, and systems of equations, and 3) representations
•
Measurement reasoning, including 1) measurement involving physical and
natural attributes, 2) systems of measurement and 3) measurement involving
geometry and algebra
•
Probabilistic reasoning, including 1) counting principles and 2) computation and
interpretation of probabilities
•
Functions, including recognition and representation of functions
Overall, then, the CCRS are more rigorous and demanding than the corresponding test
items, except specific mathematics performance expectations as indicated above.
Range of Knowledge
In addition to categorical concurrence, the alignment between the CCRS and the
Improving Alignment Between Postsecondary and Secondary Education
68
assessments are described using range of knowledge. The range of knowledge
correspondence criterion examines the alignment of assessment items to expectations
within the CCRS. It describes how a breadth of knowledge the standards expect of
students corresponds to knowledge needed to correctly answer test items. Range of
knowledge is calculated as the percent of organizing components that have at least one
item assessing each performance expectation. This is summarized across reviewers,
and the suggested benchmark is for a test to have one or more items assess at least
half of the performance expectations within an organizing component.
Table 21 describes the percent of performance expectations that are assessed by at
least one mathematics test item (summed across all reviewers). Recall that each test
item assesses more than one performance expectation. On average, across all key
content, the mathematics tests covered between 58 to 81 percent of the content
described by the CCRS.
Table 21. Range of Knowledge (Using Total Number of Matches) by Mathematics Test
Key Content
Organizing Component
Percent of Mathematics Performance Expectations
Assessed By At Least One Test Item
A
Numeric
Reasoning
Algebraic
Reasoning
Geometric
Reasoning
B
C
D
E
F
A. Number representation
50%
50%
100%
100%
100%
50%
B. Number operations
100%
100%
100%
100%
100%
100%
C. Number sense and
number concepts
100%
100%
100%
100%
100%
100%
Numeric reasoning Overall
83%
83%
100%
100%
100%
83%
A. Expressions and
equations
100%
100%
100%
100%
100%
100%
B. Manipulating expression
100%
100%
100%
100%
100%
100%
C. Solving equations,
inequalities, and systems
of equations
100%
50%
100%
100%
100%
50%
D. Representations
100%
100%
100%
100%
100%
100%
Algebraic Reasoning
Overall
100%
88%
100%
100%
100%
88%
A. Figures and their
properties
100%
100%
100%
100%
100%
100%
0%
66%
33%
33%
100%
33%
B. Transformations and
symmetry
Improving Alignment Between Postsecondary and Secondary Education
69
Key Content
Measurement
Reasoning
Probabilistic
Reasoning
Statistical
Reasoning
Functions
Problem Solving
and Reasoning
Organizing Component
Percent of Mathematics Performance Expectations
Assessed By At Least One Test Item
A
B
C
D
E
F
C. Connections between
geometry and other
mathematical content strands
66%
66%
66%
66%
100%
66%
D. Logic and reasoning in
geometry
0%
100%
100%
100%
100%
100%
Geometric Reasoning
Overall
42%
83%
75%
75%
100%
75%
A. Measurement involving
physical and natural
attributes
100%
0%
66%
100%
100%
100%
B. Systems of measurement
0%
50%
100%
100%
100%
100%
C. Measurement involving
geometry and algebra
66%
66%
67%
100%
100%
100%
D. Measurement involving
statistics and probability
0%
0%
0%
0%
50%
50%
Measurement Reasoning
Overall
42%
29%
58%
75%
88%
88%
A. Counting principles
0%
0%
100%
100%
100%
0%
B. Computation and
interpretation of probabilities
0%
0%
0%
50%
100%
0%
Probabilistic Reasoning
Overall
0%
0%
50%
75%
100%
0%
A. Data collection
0%
0%
0%
0%
0%
0%
B. Describe data
0%
25%
50%
75%
75%
100%
C. Read, analyze, interpret,
and draw conclusions
from data
0%
0%
50%
50%
100%
100%
Statistical Reasoning
Overall
0%
13%
33%
42%
58%
67%
A. Recognition and
representation of functions
50%
0%
50%
100%
50%
50%
B. Analysis of functions
100%
100%
100%
100%
100%
100%
C. Model real world situations
with functions
50%
100%
100%
100%
100%
Functions Overall
67%
67%
83%
100%
83%
75%
A. Mathematical problem
solving
80%
80%
80%
100%
80%
80%
B. Logical reasoning
0%
50%
100%
100%
100%
100%
C. Real world problem
solving
33%
100%
33%
100%
100%
100%
Problem Solving
and reasoning Overall
38%
77%
71%
100%
93%
93%
Improving Alignment Between Postsecondary and Secondary Education
70
Key Content
Communication
and
Representation
Connections
Key Cognitive
Strategies
Foundational Skills
Overall
Organizing Component
Percent of Mathematics Performance Expectations
Assessed By At Least One Test Item
A
B
C
D
E
F
A. Language, terms, and
symbols of mathematics
66%
100%
100%
100%
100%
100%
B. Interpretation of
mathematical work
50%
100%
50%
100%
100%
100%
C. Presentation and
representation of
mathematical work
0%
0%
33%
33%
33%
100%
Communications and
Representations Overall
39%
67%
61%
78%
78%
100%
A. Connections among the
strands of mathematics
50%
100%
50%
100%
50%
100%
B. Connections of
mathematics to nature, realworld situations, and
everyday life
33%
66%
33%
33%
33%
100%
Connections Overall
42%
83%
42%
67%
42%
100%
A. Intellectual curiosity
0%
0%
50%
0%
0%
0%
B. Reasoning
0%
0%
0%
25%
0%
25%
C. Problem solving
66%
66%
67%
67%
67%
100%
D. Academic behaviors
0%
0%
0%
0%
25%
0%
E. Work habits
0%
0%
0%
0%
0%
0%
F. Academic integrity
0%
0%
0%
0%
0%
0%
Key Cognitive Strategies
Overall
11%
11%
20%
15%
15%
21%
A. Reading across the
curriculum
12%
12%
38%
25%
25%
25%
B. Writing across the
curriculum
0%
0%
0%
0%
0%
0%
C. Research across the
curriculum
0%
0%
0%
0%
0%
0%
D. Use of data
0%
0%
0%
0%
33%
100%
E. Technology
0%
0%
0%
0%
0%
0%
Foundational Skills Overall
2%
2%
8%
5%
12%
25%
Total
39%
50%
58%
69%
72%
68%
Note: Percents were computed using the total number of matches (if different reviewers identified the same item as
assessing a performance expectation, that item counted each time it was identified). Shaded cells indicate the
recommended benchmark of 50 percent was met.
Using the criteria that a test have at least one item assessing each performance
expectation, all the tests sufficiently assess numeric reasoning, algebraic reasoning,
geometric reasoning, measurement reasoning, functions, problem solving and
Improving Alignment Between Postsecondary and Secondary Education
71
reasoning, and communication and representation.
Less well covered are the cross-disciplinary skills. None of the tests sufficiently assess
this, although all tests do asses the problem solving component of the key cognitive
strategies, and a single test assesses the use-of-data component of the foundational
skills and has the best overall assessment of foundational skills.
The ELA content is less consistently assessed by the tests than the mathematics
content. Table 22 describes the range of knowledge for the ELA. On average, across all
key content, the tests covered between 34 to 62 percent of the content described by the
CCRS.
Table 22. Range of Knowledge (using total number of matches) by ELA Test
Key Content
Writing
Reading
Research
Organizing Component
Percent of ELA Performance Expectations
Assessed by at Least One Test Item
A
B
C
D
E
F
A. Compose a variety of texts that
demonstrate clear focus, the logical
development of ideas in well-organized
paragraphs, and the use of appropriate
language that advances the author's
purpose.
100%
100%
100%
100%
100%
100%
Writing Overall
100%
100%
100%
100%
100%
100%
A. Locate explicit textual information and
draw complex inferences, analyze, and
evaluate the information within and across
texts of varying lengths.
73%
73%
55%
45%
64%
64%
B. Understand new vocabulary and
concepts and use them accurately in
reading, speaking, and writing.
0%
33%
33%
33%
33%
33%
C. Describe, analyze, and evaluate
information within and across literary and
other texts from a variety of cultures and
historical periods.
0%
50%
25%
0%
0%
0%
D. Explain how literary and other texts evoke
personal experience and reveal character in
particular historical circumstances.
0%
50%
0%
0%
0%
0%
Reading Overall
18%
52%
28%
20%
24%
24%
0
0
0
0
0
0
A. Formulate topic and questions.
Improving Alignment Between Postsecondary and Secondary Education
72
Key Content
Key Cognitive
Strategies
Foundational
Skills
Overall
Organizing Component
Percent of ELA Performance Expectations
Assessed by at Least One Test Item
A
B
C
D
E
F
B. Select information from a variety of
sources.
0%
0%
0%
0%
75%
0%
Research Overall
0%
0%
0%
0%
38%
0%
A. Intellectual curiosity
0%
0%
0%
100%
100%
0%
B. Reasoning
100%
75%
25%
100%
100%
75%
C. Problem solving
67%
33%
33%
100%
67%
33%
D. Academic behaviors
25%
25%
25%
100%
50%
25%
E. Work habits
0%
0%
0%
100%
0%
0%
F. Academic integrity
25%
0%
0%
100%
50%
25%
Key Cognitive Strategies Overall
36%
22%
14%
100%
61%
26%
A. Reading across the curriculum
63%
88%
75%
75%
75%
75%
B. Writing across the curriculum
67%
100%
67%
100%
100%
67%
C. Research across the curriculum
0%
0%
0%
100%
0%
0%
D. Use of data
0%
0%
0%
100%
0%
33%
E. Technology
0%
0%
0%
75%
0%
0%
Foundational Skills Overall
26%
38%
28%
90%
35%
35%
Total
36%
42%
34%
62%
52%
37%
Note: Percents were computed using the total number of matches (so that if different reviewers identified the same
item as assessing a performance expectation, that item counted each time it was identified). Shaded cells in the table
indicate the recommended benchmark of 50 percent was met.
While categorical concurrence identifies the tests that have enough items to reliably
assess individual performance expectations, range of knowledge identifies the tests that
assess the most performance expectations. Using the criteria that tests have at least
one item assessing each performance expectation, all tests sufficiently assess writing
only. Reading and the cross-disciplinary skills are less well covered by the tests. As with
mathematics, the organizing components high in the range of knowledge tended to be
high across all tests, regardless of the number of items. When these components were
low, they tended to be low across all tests.
Interpretation of Findings
Results suggest that most of the admissions and placement tests reviewed align to a
moderate to high level with the CCRS performance expectations in mathematics and
Improving Alignment Between Postsecondary and Secondary Education
73
ELA. None of the tests aligns at a high level with the cross-disciplinary skills. The tests
exhibited similar results for categorical-concurrence and range-of-knowledge measures.
In mathematics, alignment (as measured by both categorical concurrence and range of
knowledge) is highest for numeric reasoning, algebraic reasoning, functions, problem
solving and reasoning, and communications and representations; lower for connections,
geometric reasoning, measurement reasoning, and statistical reasoning; and lowest for
foundational skills, probabilistic reasoning, and key cognitive strategies. These findings
are consistent with the tests specifications. All of the tests are designed to assess
numeric reasoning, algebraic reasoning, geometric reasoning, and functions; only onethird are designed to assess measurement reasoning, probabilistic reasoning, statistical
reasoning, and problem solving reasoning; and none of the tests specify the
connections content that are in the CCRS.
In ELA, alignment (as measured by categorical concurrence and range of knowledge) is
highest for writing, lower for reading and foundational skills, and lowest for key cognitive
strategies. Although minimally assessed by one test, none of the tests provide sufficient
assessment of the research performance expectations contained in the CCRS. These
findings are somewhat unexpected because all ELA tests are designed to assess
reading and writing, yet only writing shows consistent and sufficient alignment.
Rater agreement was higher for mathematics items and standards than for ELA items
and standards, suggesting perhaps that college readiness standards are still a new
phenomenon for reviewers and that special attention needs to be paid to developing
stronger common mental maps of the standards on the part of all reviewers before they
begin rating test items. Reviewer agreement was higher for cognitive-demand ratings
than for rigor ratings, which is counterintuitive as rigor is rated on a smaller scale (1–3
rather than 1–4) and evaluating whether an item or a standard is at the level of an entrylevel college student might appear to be a more straightforward distinction to make than
determining the thought process required by an item or standard. Other researchers
Improving Alignment Between Postsecondary and Secondary Education
74
report similar findings (Herman, Webb, Zuniga, 2005), suggesting that it may be easier
to apply cognitive-demand ratings consistently than it is to apply rigor ratings
consistently. While an agreement of 0.8 or higher is ideal, other studies in this area have
yielded typical agreement measures for alignment studies closer to 0.5 (Blank, 2007), a
criterion level that this study exceeds.
Analysis of the rigor and cognitive-demand ratings shows similar patterns for both
measures. For the CCRS subject areas, the ELA performance expectations are more
rigorous and cognitively demanding than are the cross-disciplinary skills and the
mathematics standards. The mathematics standards are the least rigorous, averaging
below the level at which an entry-level college student should be expected to perform.
The ELA standards are the highest and averaged close to the level at which entry-level
college students should be expected to perform. The cross-disciplinary skills fall
between mathematics and ELA.
The most rigorous and cognitively demanding mathematics concepts involve statistical
and logical reasoning. Although statistics is increasingly taught in high school, its
concepts may be more abstract than typical high school mathematics content. Logical
reasoning is a complex skill that is difficult to assess with multiple-choice items. The
CCRS are more rigorous and cognitively demanding than the test items in this area.
This should not be unexpected because the CCRS describe the knowledge and skills
that college-ready students should have, while the test items assess a much broader
range of proficiency in order to determine the level necessary to enter a credit-bearing
course.
Some CCRS are not assessed by any items on any test. For mathematics, these
include skills that may not be assessable by multiple-choice test items, including
practical mathematics skills (planning a study, collecting data, analyzing relationships
between paired data, recognizing reliability of statistical results). For ELA, these include
practical reading skills (using reference guides to confirm the meanings of new words or
Improving Alignment Between Postsecondary and Secondary Education
75
concepts, reading a variety of texts from world literatures, etc.) and practical research
skills (formulating a research question, exploring a research topic, using source material
ethically, etc.) and include skills not practical in a typical standardized testing
environment, even with the use of constructed-response type items.
The lack of coverage for these standards suggests that to assess the full range of
knowledge and skills necessary to be college ready in the state of Texas may eventually
require additional methods. Previous research supports this, suggesting that item type is
an important factor in determining what assessable content is (Rothman, 2004), and
that using multiple methods, including performance assessment and constructed
response-type items in more subjects than writing, may be necessary to cover fully
content that is not assessable through multiple-choice items. It may be worth
considering combining multiple-choice test results with end-of-course-exam results,
senior demonstrations or projects, performance assessments, or portfolios in order to
gauge student readiness across the full set of college and career standards.
Additional work may be needed to standardize the methodology for including test pools
in alignment studies. As noted earlier, previous researchers using both test forms and
item pools in an alignment study have tended not to address explicitly the likely impact
on findings resulting from the different number of items. Although similar results were
found across test pools and forms, metrics that rely upon counts may not always be
directly comparable. Future research should be conducted to refine methods for direct
comparisons of test forms to test-item banks.
Finally, based on this study, the recommendation is to consider the development and
introduction of measures and assessments that are based on broader and more robust
conceptions of college readiness that are consistent with the full set of Texas College
and Career Readiness Standards and a fuller set of college readiness expectations.
Such an approach would help lead the nation in a direction in which college readiness
became the expression of a full range of skills and capabilities, some of which can be
Improving Alignment Between Postsecondary and Secondary Education
76
measured by current commercially available instruments and others that will require
new tools and methods. Such a system would be able to provide the full range of
information that is necessary to know if students are truly college ready and to identify
and diagnose the specific areas where students need additional help, practice, support,
and skill building to be prepared to succeed in entry-level college courses.
Improving Alignment Between Postsecondary and Secondary Education
77
References
Achieve, I. (2006). Closing the Expectation Gap. Washington, DC: Author.
Achieve. (2007). Aligned expectations? A closer look at college admissions and
placement tests. Washington, DC: Achieve, Inc.
ACT. (2005). Crisis at the core: Preparing all students for college and work access.
Iowa City, IA: Act, Inc.
ACT. (2009a). Retrieved March 13, 2009, from http://www.act.org/aap/index.html
ACT. (2009b). Retrieved March 13, 2009, from http://www.act.org/ASSET/
ACT. (2009c). Retrieved March 13, 2009, from http://www.act.org/COMPASS/index.html
Brown, R. S., & Conley, D. (2007). "Comparing State High School Assessments to
Standards for Success in Entry-Level University Courses." Journal of Educational
Assessment, 12(3), 137-160.
Conley, D.T. (2003a). Mixed messages: What state high school tests communicate
about student readiness for college. Eugene: University of Oregon, Center for
Educational Policy Research.
Conley, D.T. (2003b). Understanding university success. Eugene: University of Oregon,
Center for Educational Policy Research.
Conley, D. (2003). Who Governs Our Schools? Changing Roles and Responsibilities.
New York: Teachers College Press.
Conley, D. (2005). College Knowledge: What It Really Takes for Students to Succeed
and What We Can Do to Get Them Ready. San Francisco: Jossey-Bass.
Improving Alignment Between Postsecondary and Secondary Education
78
Conley, D. (2007). Redefining college readiness. Eugene, OR: Educational Policy
Improvement Center.
DʼAgostino, J.V., Welsh, M., Cimetta, A., Falco, L., Smith, S., VanWinkle, W., and
Powers, S. (2008). “The rating and matching item-objective alignment methods.”
Applied Measurement in Education, 21, 1–21.
Fuhrman, S. H. (Ed.). (1993). Designing Coherent Education Policy: Improving the
System. San Francisco: Jossey-Bass.
Fuhrman, S. H., & Elmore, R. F. (Eds.). (2004). Redesigning Accountability Systems for
Education. New York: Teachers College Press.
Herman, J. L., Webb, N. M., and Zuniga, S. A. (2007). “Measurement issues in the
alignment of standards and assessments: A Case study.” Applied Measurement
in Education, 20,101–126.
Herman, J. L., and Webb, N. M. (Eds.) (2007). Special Issue of Applied Measurement in
Education: Alignment Issues, 20, 1–135.
Musquash, C., and OʼConnor, B. (2006). “SPSS and SAS programs for generalizability
theory analyses.” Behavior Research Methods, 38 (3), 542–547.
National Center for Education Statistics [NCES] (2004). The condition of education
2004, indicator 18: Remediation and degree completion. Washington, DC: U.S.
Department of Education.
National Center for Educational Statistics (2007). The Condition of Education 2007.
Washington, D.C.: U.S. Department of Education.
Improving Alignment Between Postsecondary and Secondary Education
79
Porter, A. C. (2002). “Measuring the content of instruction: Uses in research and
practice.” Educational Researcher, 31(7), 3–14.
Rothman, R. (2003). Imperfect matches: The alignment of standards and tests. National
Research Council.
Shavelson, R. J., and Webb, N. M. (1991). Generalizability theory: A primer. Newbury
Park, CA: Sage Publications.
Thompson, B. (2003). A brief introduction to generalizability theory. In B. Thompson,
(Ed.) Score reliability (pp. 43–58). Thousand Oaks, CA: SAGE Publications.
Webb, N.L. (1997). Criteria for alignment of expectations and assessments in
mathematics and science education (Research Monograph No. 6). Washington,
DC: Council of Chief State School Officers.
Webb, N.L. (1999). Alignment of science and mathematics standards and assessments
in four states (Research Monograph No. 18). Madison: University of Wisconsin,
Wisconsin Center for Education Research.
Webb, N. L. (2002, April). An analysis of the alignment between mathematics standards
and assessments for three states. Paper presented at the annual meeting of the
American Educational Research Association, New Orleans.
Improving Alignment Between Postsecondary and Secondary Education
80
Appendix A: Texas College and Career Readiness CrossDisciplinary Standards
The following cross-disciplinary standards are excerpted from the Texas College and
Career Readiness Standards (CCRS). To view all CCRS, including the
English/language arts, mathematics, science, and social standards, please go to:
http://www.thecb.state.tx.us/collegereadiness/TCRS.cfm
The cross-disciplinary standards are organized into two major areas: Key Cognitive
Skills and Foundational Skills. The Key Cognitive Skills specify intellectual behaviors
that are prevalent in entry-level college courses. The list includes intellectual curiosity,
reasoning, problem solving, academic behaviors, work habits, and academic integrity.
Foundational Skills consist of proficiencies students need to be able to transfer
knowledge and apply it across the curriculum. These include reading, writing,
conducting research, understanding and using data, and using technology.
The first three levels of the cross-disciplinary standards are written to apply across
subject areas. The performance indicators found in the appendix illustrate how the
cross-disciplinary standards are manifested within the subject areas. The Vertical
Teams created an example in each subject area of at least one performance indicator
that could be applied in that subject area. These indicators are meant to exemplify how
the cross-disciplinary standards could be demonstrated in all subject areas.
I. Key Cognitive Skills
A. Intellectual curiosity
1. Engage in scholarly inquiry and dialogue.
a. Identify what is known, not known, and what one wants to know in a
problem.
b. Conduct investigations and observations.
c. Cite examples or illustrations in which a clear-cut answer cannot be
reached.
2. Accept constructive criticism and revise personal views when valid
evidence warrants.
a. Articulate a point of view and provide valid evidence to support findings.
b. Demonstrate willingness to take intellectual risks by investigating novel,
controversial, or unpopular opinions or conclusions.
c. Examine alternative points of view, taking different roles to defend, oppose,
and remain neutral on issues.
d. Recognize conflicting information or unexplained phenomena.
Improving Alignment Between Postsecondary and Secondary Education
81
B. Reasoning
1. Consider arguments and conclusions of self and others.
a. Know and apply logic to analyze patterns and descriptions and to evaluate
conclusions.
b. Cite valid examples or illustrations that support the conclusions.
c. Question whether the claims and conclusions of self and others are
supported by evidence.
d. Identify counter examples to disprove a conclusion.
2. Construct well-reasoned arguments to explain phenomena, validate
conjectures, or support positions.
a. Participate in a debate that is based on facts and has a logical structure.
b. Construct a visual presentation, including hypothesis, data, results, and
conclusion.
c. Write a paper that addresses counter-arguments to advocated positions.
d. Recognize and apply techniques of statistical or probabilistic analysis to
judge reliability of information.
e. Organize an argument separating fact from opinion.
3. Gather evidence to support arguments, findings, or lines of reasoning.
a. Use different kinds of data (e.g., case studies, statistics, surveys,
documents) to support an argument.
b. Evaluate evidence in terms of quality and quantity.
c. Describe limitations of data collection methods.
4. Support or modify claims based on the results of an inquiry.
a. Refine claims and adjust a position in response to inquiry.
b. Review and check strategies and calculations, using alternative approaches
when possible.
C. Problem solving
1. Analyze a situation to identify a problem to be solved.
a. Represent and/or restate the problem in one or more ways (e.g., graph,
table, equation), showing recognition of important details and significant
parameters.
b. Break complex problems into component parts that can be analyzed and
solved separately.
c. Apply previously learned knowledge to new situations.
d. Analyze a media report, identify any misuse of statistics, and suggest ways
to more accurately depict this information.
2. Develop and apply multiple strategies to solve a problem.
a. Use a range of standard methods, devices, techniques, and strategies to
gather and analyze information.
b. Use knowledge gained from other subject areas to solve a given problem.
3. Collect evidence and data systematically and directly relate to solving a
problem.
Improving Alignment Between Postsecondary and Secondary Education
82
a. Use general and specialized reference works and databases to locate
sources.
b. Collect evidence and data directly related to solving the problem and
eliminate irrelevant information.
c. Produce charts, graphs, and diagrams accurately, including scale, labeling,
units, and organization.
d. Present the collected data visually, describe the data collection procedure,
and defend choosing that procedure over other possibilities.
D. Academic behaviors
1. Self-monitor learning needs and seek assistance when needed.
a. Ask questions to check for understanding or to clarify information.
b. Use a systematic method for recording, storing, and organizing materials and
resources; avoid haphazard or messy accumulation of information.
2. Use study habits necessary to manage academic pursuits and
requirements.
a. Manage time effectively to complete tasks on time.
b. Demonstrate accurate note-taking.
c. Use the appropriate level of detail necessary to complete an assigned task.
d. Balance academic and non-academic activities to successfully participate in
both.
3. Strive for accuracy and precision.
a. Collect and report experimental data carefully and correctly.
b. Produce charts, graphs, and diagrams accurately, including scale, labeling,
units, and organization.
c. Eliminate irrelevant information from an assignment.
4. Persevere to complete and master tasks.
a. Persevere until a task is completed by working even when faced with
uncertainty or open-ended assignments.
b. Seek assistance when needed to complete the assignment.
c. Recognize when a task is completed.
E. Work habits
1. Work independently.
a. Plan a project, establish its parameters, and complete it with minimal
supervision, seeking assistance accordingly.
b. Follow directions or procedures independently.
c. Complete assignments outside the classroom setting in a timely manner.
2. Work collaboratively.
a. Work collaboratively with students from various cultural and ethnic
backgrounds.
b. Distinguish between situations where collaborative work is appropriate and
where it is not.
Improving Alignment Between Postsecondary and Secondary Education
83
c. Work in small groups to investigate a problem or conduct an experiment.
F. Academic integrity
1. Attribute ideas and information to source materials and people.
a. Document the work of others, giving credit where credit is due and never
claim credit for work that is not oneʼs own.
b. Use standard bibliographic and reference citation formats, choosing the
style appropriate to the subject and the audience.
c. Define plagiarism and articulate the consequences of academic dishonesty.
2. Evaluate sources for quality of content, validity, credibility, and relevance.
a. Verify validity of a source within a submitted work.
b. Compare and contrast coverage of a single topic from multiple media
sources.
3. Include the ideas of others and the complexities of the debate, issue, or
problem.
a. Present multiple perspectives of an issue.
b. Represent accurately the data, conclusions, or opinions of others.
4. Understand and adhere to ethical codes of conduct.
a. Follow copyright laws and restrictions.
b. Use technology responsibly (e.g., avoiding malice, misrepresentation, or
misleading use of information).
II. Foundational Skills
A. Reading across the curriculum
1. Use effective prereading strategies.
a. Use the title, knowledge of the author, and place of publication to make
predictions about a text.
b. Use a table of contents to preview a text and understand its design.
c. Scan headline sections or other division markers, graphics, or sidebars to
form an overview of a text.
2. Use a variety of strategies to understand the meanings of new words.
a. Use context clues, including definitions, examples, comparison, contrast,
cause and effect, and details provided in surrounding text.
b. Consult references (e.g., dictionary, thesaurus) effectively.
c. Understand notation specific to discipline (e.g., mathematical notation,
scientific symbols).
3. Identify the intended purpose and audience of the text.
a. Predict purpose and audience of a text based on the title, preface, and other
features of a text.
b. Explain how the language of an effective text targets an intended audience.
c. Explain the importance of a technical and/or scientific article.
Improving Alignment Between Postsecondary and Secondary Education
84
4. Identify the key information and supporting details.
a. Outline a chapter of an informational text.
b. Summarize the major points in a text, and use graphic organizers (e.g.,
concept maps, diagrams) to organize ideas and concepts in a visual
manner.
c. Analyze connections between major and minor ideas.
d. Identify and define key terminology from technical and/or scientific
documents.
5. Analyze textual information critically.
a. Identify faulty premises in an argument.
b. Identify stated and implied assumptions.
c. Identify conclusions unsupported by sufficient evidence in informational
texts.
d. Use inductive and deductive reasoning.
e. Draw conclusions based on evidence, support, or data through logical
reasoning.
f. Compare a primary source and an interpretation in a textbook.
6. Annotate, summarize, paraphrase, and outline texts when appropriate.
a. Outline an informational or literary text.
b. Annotate text for comprehension and analysis.
c. Summarize an article to demonstrate comprehension.
d. Paraphrase a writerʼs ideas or findings.
7. Adapt reading strategies according to structure of texts.
a. Identify a variety of textual forms and genres (e.g., long and short texts) and
adapt reading strategies accordingly.
b. List strategies to use during reading, including:
• Anticipate and predict what information the text is likely to contain.
• Monitor understanding by self-questioning.
• Use strategies (e.g., mental imagery, paraphrasing, information in
glossaries) to re-examine the text if comprehension fails.
• Reread difficult passages.
• Read ahead for additional clarification.
• Seek assistance for clarification.
• Self-monitor and summarize the information gained.
c. Explain how form or genre communicates meaning.
8. Connect reading to historical and current events and personal interest.
a. Locate an article or source that relates to a class topic and explain the
relevance.
B. Writing across the curriculum
1. Write clearly and coherently using standard writing conventions.
a. Prepare a topic proposal that specifies a purpose and justifies the choice of
audience to achieve that purpose.
Improving Alignment Between Postsecondary and Secondary Education
85
b. Craft a thesis statement that articulates a position and list relevant evidence
and examples in logical groupings.
c. Use symbols, diagrams, graphs, and words to communicate ideas.
d. Use appropriate terminology and data expression to communicate
information in a concise manner.
e. Use a variety of reference guides for citation conventions, grammar,
mechanics, and punctuation.
2. Write in a variety of forms for various audiences and purposes.
a. Present an argument supported by relevant evidence, examples, and
counterarguments.
b. Prepare a summary or abstract of a journal article or report, extracting in brief
form the pertinent information.
c. Evaluate articles by analyzing the study design, data source, graphical
representation of data, and analyzed data results reported (or not reported).
d. Write a reflection about the process selected to conduct research or solve a
problem.
e. Write accurate and understandable lab reports and technical documents.
3. Compose and revise drafts.
a. Submit a writing assignment to be proofread by a teacher, parent, or other
student. Revise the paper, incorporating constructive criticism when
appropriate.
b. Edit text for correct spelling, capitalization, and punctuation.
c. Edit for appropriate tense and voice.
d. Edit for correct word use.
e. Use a variety of reference guides for citation conventions, grammar,
mechanics, and punctuation.
f. Submit a final draft that is easily read and has few or no grammatical or
spelling errors.
C. Research across the curriculum
1. Understand which topics or questions are to be investigated.
a. Formulate research questions.
b. Use strategies like those in the writing process to generate questions and
areas to pursue.
c. Consult previous studies or conduct interviews with experts to identify
questions central to a research topic.
d. Propose explicit, testable hypotheses, using the “if ..., then ...” format.
2. Explore a research topic.
a. Produce an annotated list of sources consulted, differentiating among
primary, secondary, and other sources and explain their relevance to the
research topic.
b. Outline the most significant controversies or questions on a research topic.
c. Plan an investigative study.
Improving Alignment Between Postsecondary and Secondary Education
86
d. Explain reasons for valid competing points of view on a given topic.
3. Refine research topic based on preliminary research and devise a timeline
for completing work.
a. Gather information from a variety of relevant sources.
b. Use general and specialized reference works and databases to locate
sources.
c. Locate electronic sources, when appropriate, using advanced search
strategies.
d. Select an appropriate range of source materials.
e. Analyze a wide range of sources, including technical texts, primary and
secondary sources, conflicting points of view, and interdisciplinary research
when appropriate.
f. Design and carry out hands-on experimental investigations, choosing
appropriate apparatuses, identifying controls and variables, tentatively
predicting the outcome of the procedures, and evaluating whether actual
results agree with predicted results.
g. Use numerical and mathematical tools such as software, including
databases, spreadsheets, and other tools, in investigations and
explanations.
4. Evaluate the validity and reliability of sources.
a. State explicitly characteristics or identifying features that indicate accuracy
or reliability of sources, to determine whether sources are biased,
incomplete, or otherwise unreliable.
b. Follow a set of criteria to determine the validity and reliability of sources.
c. Identify claims found in one or more of the sources that require support or
verification, and evaluate the informationʼs validity.
d. Evaluate the data presented in graphics, tables, charts, and maps when
appropriate to the topic.
5. Synthesize and organize information effectively.
a. Select quotations and evidence that support the thesis.
b. Determine what evidence best supports conclusions.
c. Use well-organized strategies to collect and organize information gathered.
d. Determine the best order for presenting evidence that supports conclusions.
6. Design and present an effective product.
a. Determine the best order for presenting major and minor points.
b. Design a report using features such as headings and graphics appropriate
to the writing task.
c. Use a citation system specified by or appropriate to the assignment.
7. Integrate source material.
a. Integrate source material into text by a combination of accurately
summarizing, paraphrasing, and quoting.
b. Balance use of source material with relevant explanations.
c. Use source material ethically.
Improving Alignment Between Postsecondary and Secondary Education
87
d. Understand and avoid all types of plagiarism.
8. Present final product.
a. Use appropriate media for presentation of research results.
b. Document sources using a standard format appropriate to the subject area.
D. Use of data
1. Identify patterns or departures from patterns among data.
a. Identify patterns from multiple representations of data such as graphical and
tabular forms.
b. Review current news events and evaluate possible connections (e.g., linking
economic data with political events).
2. Use statistical and probabilistic skills necessary for planning an
investigation and collecting, analyzing, and interpreting data.
a. Create representations of data (e.g., data tables, correctly labeled and
scaled graphs, narrative descriptions).
b. Evaluate a given published report for missing information and misuse of
data.
3. Present analyzed data and communicate findings in a variety of formats.
a. Compose a written document detailing a research project.
b. Use appropriate visuals and statistical results to convey findings to a
specified audience.
E. Technology
1. Use technology to gather information.
a. Use the Internet or other appropriate technologies to post survey questions
on an assigned topic.
b. Use devices to measure physical properties.
c. Use online databases to access scholarly work on an assigned research
topic.
2. Use technology to organize, manage, and analyze information.
a. Use data analysis software to analyze survey results.
b. Use spreadsheets to manage and organize statistical data.
3. Use technology to communicate and display findings in a clear and
coherent manner.
a. Create spreadsheets and graphs to communicate findings in a presentation
that includes graphics, visuals, or other supporting images.
b. Utilize technology to present information and/or data in a variety of ways.
4. Use technology appropriately.
a. Explain how technology is a useful and effective tool to communicate
findings.
b. Identify when technology may not be necessary or appropriate to
communicate findings.
c. Formulate strategies to communicate findings with and without technology.
Improving Alignment Between Postsecondary and Secondary Education
88
Appendix B: Scale Items & Rationale Statements
for Examining the Alignment between the Texas College
and Career Readiness Standards and Entry-Level College
Courses at Texas Postsecondary Institutions
Most Necessary for Preparation to Succeed in this Course
• This element is critical for success in the course
• Course is taught with the assumption that students already know this information
• This element will not be retaught in this course
• Students will have difficulty succeeding in the course if they have not learned this
element previously
• Other – provide reason
More Necessary for Preparation to Succeed in this Course
• This element is important for success in the course
• Course is taught with the assumption that students are at least familiar with or
aware of this element
• This element will be reviewed only and not retaught in this course
• Students will benefit from having learned this element previous to the course but
can probably relearn it during the course and still succeed in the course
• Other – provide reason
Less Necessary for Preparation to Succeed in this Course
• Student knowledge of and familiarity with this element may be helpful
• Course is taught with the assumption that students are familiar with the element
on a very general level
• This element will be taught in some detail in the class
• Even if students have not learned this previously, they will be able to learn it in
the course at a level sufficient to succeed in the course
• Other – provide reason
Least Necessary for Preparation to Succeed in this Course
• Students need only minimal knowledge of and familiarity with this element
• Course is taught with the assumption that students may be only vaguely aware of
the element
• This element will be taught as new material in this course even if students have been
taught it before
• Students will be able to succeed in this course even if they only have a very general
awareness or understanding of this element when they enter the course
• Other – provide reason
Improving Alignment Between Postsecondary and Secondary Education
89
Not Necessary for Preparation to Succeed in this Course
• This element is too advanced for this course
• This element will be encountered for the first time in subsequent courses in the
subject area
• This element is too specialized or specific for this course
• This element is irrelevant to this course
• This element will be introduced as new material in this course with the assumption
that students have not learned anything about it before this course
• Other – provide reason
Improving Alignment Between Postsecondary and Secondary Education
90
Appendix C: Reference Course Profile:
English Composition II
ENGL 1302
Principles and techniques of written, expository, and persuasive composition; analysis
of literary, expository, and persuasive texts; and critical thinking.*
Reference Course Profile: Overview and Purpose
A Reference Course Profile represents current practice in entry-level college courses in
Texas. Twenty-six course profiles were developed from an extensive study of over 900
entry-level general education and CTE college courses offered at Texas institutions of
higher education. Each course profile consists of a course description, sample syllabus,
reference lists, attendant course materials (such as assignments, assessments, and, in some
cases, scoring rubrics), and the Texas College and Career Readiness Standards addressed
within the course.
The Profiles provide a reference point for college readiness that fosters increased
transparency between secondary and postsecondary education. They are not intended to
be mandatory, prescriptive, or best practice.
Prerequisites and Prior Knowledge
Most college-level syllabi do not list all prior knowledge required to succeed in the
course; it is usually an underlying assumption. For the purpose of this Reference Course
Profile, the required prior knowledge and skills students need to be successful in the
course are explicitly stated to help both secondary and postsecondary faculty in
establishing goals and expectations for their students. The knowledge and skills reflected
in the outline in this section are pulled directly from the Texas College and Career
Readiness Standards (CCRS), written and validated by Texas faculty in 2007-8. The CCRS
are available online at: http://www.thecb.state.tx.us/collegereadiness/CRS.pdf
Prior to enrolling in this course, students must
• Satisfy Texas Success Initiative (TSI) requirements set by the institution as
described in Coordinating Board rule (Texas Administrative Code, Chapter 4,
Subchapter C).
• ENGL 1301 or its equivalent.
*
From the course description appearing in the Lower Division Academic Course Guide Manual (ACGM)
http://www.thecb.state.tx.us/AAR/UndergraduateEd/WorkforceEd/acgm.htm
Improving Alignment Between Postsecondary and Secondary Education
91
In addition, students should have the following College and Career Readiness
Standards skills. Only the specific standards and performance expectations pertinent to
the course are listed on the following pages.
English/Language Arts College and Career Readiness Standards
I. Writing
B. Compose a variety of texts that demonstrate clear focus, the logical
development of ideas in well-organized paragraphs, and the use of
appropriate language that advances the authorʼs purpose.
II. Reading
A. Locate explicit textual information, draw complex inferences, and analyze and
evaluate the information within and across texts of varying lengths.
B. Understand new vocabulary and concepts and use them accurately in
reading, speaking, and writing.
III. Speaking
A. Understand the elements of communication both in informal group discussion
and formal presentations.
B. Develop effective speaking styles for both group and one-on-one situations.
IV. Listening
A. Apply listening skills as an individual and as a member of a group in a variety
of settings.
B. Listen effectively in informal and formal situations.
V. Research
A. Formulate topic and questions.
B. Select information from a variety of sources.
Cross-Disciplinary Standards
I. Key Cognitive Skills
A. Intellectual Curiosity
B. Reasoning
C. Problem Solving
D. Academic Behaviors
E. Work Habits
F. Academic Integrity
II. Foundational Skills
A. Reading Across the Curriculum
B. Writing Across the Curriculum
C. Research Across the Curriculum
D. Use of Data
E. Technology
Improving Alignment Between Postsecondary and Secondary Education
92
Course Objectives
Course objectives include the course-specific skills and knowledge that students will
possess upon completion of the course. They assist postsecondary faculty in clarifying the
goals of their courses and provide a clear picture of the expectations students will
encounter once they begin college. This sample list of objectives was adapted from syllabi
submitted in 2008 by Texas college faculty.
Upon completion of the course the student will be able to:
1. Demonstrate critical thinking skills as evidenced by the ability to analyze facts,
synthesize factual information, and evaluate opinions in light of the facts
presented throughout this course.
2. Demonstrate the ability to recognize and use both deductive and inductive logic
and to recognize logical fallacies in oneʼs own writing and in the writing of others.
3. Write a well-developed, coherent essay with a minimum of grammatical errors.
4. Utilize library resources effectively.
5. Document sources clearly while supporting a thesis with both primary and
secondary sources.
6. Demonstrate revising and editing skills.
7. Demonstrate effective speaking skills.
Research Objectives:
This course is a “research” course, which means that it intends to help students grapple
with the notion of scholarly inquiry. That is, what is research for? What are its
objectives? What are its methods? What are its shortcomings? Students should learn to
appreciate research as a highly social matter: that real research must make some kind
of contribution to the understanding of people besides just the researcher.
Critical thinking is an important objective of this course. Critical thinking is a process
involving higher order thinking skills. These skills include, but are not limited to,
application, analysis, synthesis, and evaluation of factual information.
A well-cultivated critical thinker3:
• Raises vital questions and problems, formulating them clearly and precisely.
• Gathers and assesses relevant information, using abstract ideas to interpret it
effectively.
• Comes to well-reasoned conclusions and solutions, testing them against relevant
criteria and standards.
• Thinks open-mindedly within alternative systems of thought, recognizing and
assessing their assumptions, implications, and practical consequences.
• Communicates effectively with others in finding solutions to complex problems.
3
Paul, Richard & Elder, Linda. The Miniature Guide to Critical Thinking Concepts & Tools. Dillon Beach: The
Foundation for Critical Thinking, 2005.
Improving Alignment Between Postsecondary and Secondary Education
93
Reading Objectives:
Reading at the college level means having the ability to analyze and interpret a variety
of printed materials: books, articles, and documents.
Speaking Objectives:
Effective speaking is the ability to communicate orally in clear, coherent, and persuasive
language appropriate to purpose, occasion, and audience.
Computer Literacy Objectives:
Computer literacy at the college level means having the ability to use computer-based
technology in communicating, solving problems, and acquiring information. Coreeducated students should have an understanding of the limits, problems, and
possibilities associated with the use of technology and should have the tools necessary
to evaluate and learn new technologies as they become available.
Sample Textbooks and Materials
This list is comprised of texts that appear on course syllabi submitted in 2008 by faculty
teaching entry-level college courses most representative of current practice in Texas. This
list is not exhaustive, prescriptive, or required.
• Gibaldi, Joseph. MLA Handbook for Writers of Research Papers. 6th Ed. Modern
Language Association of America, 2003.
• Hacker, Diana. A Writerʼs Reference, 6th Ed. Boston: Bedford/St. Martins, 2007.
• Lunsford, Andrea A., John J. Ruszkiewicz, and Keith Walters. Everythingʼs an
Argument (with readings). 4th Ed. Boston: Bedford/St, Martins, 2007.
• Kennedy, X. J. and Dana Gioia. Literature: An Introduction to Fiction, Poetry, and
Drama. 10th Ed. New York: Longman, 2005.
• Kress, Anne and Suellyn Winkle. Next Text: Making Connections Across and
Beyond the Disciplines. Boston: Bedford/St. Martinʼs, 2008.
• Palmquist, Mike. The Bedford Researcher. 2nd Ed. Boston: Bedford/St. Martinʼs,
2006.
• Maimon, Elaine P., Janice H. Peritz, and Kathleen Blake Yancey. The New
McGraw-Hill Handbook. Boston: McGraw-Hill, 2007.
• Mollick, Kathleen et al. The Popken Writer: A Collection of Student Research
Writing. Vol. 2. Boston: Houghton-Mifflin, 2008.
• Rottenberg, Annette T. and Donna Haisty Winchell. Elements of Argument. 9th Ed.
Boston: Bedford/St. Martins, 2009.
• Ruszkiewicz, John, Daniel E. Seward, and Maxine Hairston. SF Writer. 4th Ed.
Upper Saddle River, NJ: Prentice Hall, 2007.
• Troyka, Lynn. Simon and Schuster Handbook for Writers with i-Book. 8th Ed.
Prentice Hall. 2006.
• Wood, Nancy. Perspectives on Argument. 5th Ed. Upper Saddle River, NJ:
Prentice Hall. 2006.
Improving Alignment Between Postsecondary and Secondary Education
94
•
A good college-level desk dictionary.
Sample Methods of Instruction
Students should be prepared to encounter a variety of instructional methods, as faculty
indicate the use of several beyond the lecture format. The list of methods of instruction has
been adapted from course syllabi submitted in 2008 by faculty teaching entry-level
college courses most representative of current practice in Texas. The approximate
percentage of time allocated to each instructional method is also indicated.
1. Lecture – 20%
a. Lecture is defined as a method of instruction in which the instructor has full
responsibility for presenting material orally.
b. Lectures will take place in the form of informal lectures, in which active
student participation, such as questioning and answering, will be included.
2. Full-class Discussion of Readings and Rhetorical Ideas – 30%
a. Students will be expected to come to class ready to contribute to class
discussion.
b. Students will be expected to listen and respond to each otherʼs comments.
3. Group Discussion – 10%
a. In addition to class discussion, students will be expected to take turns
facilitating small group discussions both in and outside of class time.
4. Peer Review of Writing – 10–15%
a. Students will be expected to review the work of classmates and provide and
accept constructive feedback.
5. One-on-one Conferences with Instructor – 10–15%
a. Conferencing with the instructor on an individual basis allows students to get
critical feedback of work.
6. Library Work – 15%
a. Students are expected to become familiar with library resources and research
methods.
Sample Assignments and Assessments
A typical number of formal assignments for ENGL 1302 is four to eight essays. The course
will also include quizzes, a final exam, and a number of other assignments. Below are the
kinds of assignments that might be expected and their percentage of the final grade.
1. Writing Assignments – 55%
a. Students will complete a series of assignment culminating in a final research
paper
2. Oral Presentation of research – 5%
a. Students will be required to give a brief oral presentation of their research in
front of the class.
3. Final Exam – 20%
Improving Alignment Between Postsecondary and Secondary Education
95
4. Daily Work – 10%
a. Daily work will consist of small, in-class assignments, as well as a reading
journal that students will turn in periodically.
5. Quizzes – 10%
Sample Schedule
Samplings of assignments and assessments have been provided in the Reference Course
Profile materials. Bolded blue text indicates when a document is provided and a link to
that document is available. The time allocated for students to complete the assignment is
also indicated. The list of topics, as well as the overall pacing of the course, has been
adapted from course syllabi submitted in 2008 by faculty teaching entry-level college
courses most representative of current practice in Texas.
Week
Reading/Discussion Topics
Assignments & Assessments
1
Read Wood, Ch. 1 & 3
Read Brady pp. 59–60
Assignment 1: Reading Journal
2
Read Wood, Ch. 4. pp. 109–113
Assignment 2: Summary Connect Paper
(3–4 hours)
3
Read Wood, Ch. 5.
Turn in Reading Journal
4
Read “On Womenʼs Right to Vote” Read
Wood, “Writing an Annotated Bibliography” pp.
327–31
Assignment 3: Research Proposal (2 hours)
5
Introduce Research Project
Turn in Reading Journal
Quiz 1
6
Student/Instructor Conferences
Assignment 4: Research Paper (5–6 hours)
7
Research Project Workshop
Turn in Reading Journal (30–60 minutes)
8
Research Presentations
Wood, Ch. 6
Read Wood, Ch. 7
Assessment 1 Reader Response Rubric provided
Assignment 5: Exploratory Essay
(4–5 hours)
9
Read Wood, Ch. 8
Proofs and fallacies
Turn in Reading Journal
Quiz 2
10
Read Wood pp. 339–48 on the writing process
Read Wood, Ch. 13
Assignment 6: Position Paper (7–8 hours) or Short
Story Project (8–10 hours)
Improving Alignment Between Postsecondary and Secondary Education
96
Week
Reading/Discussion Topics
Assignments & Assessments
11
Peer review workshop
Read Wood, Ch. 9
Turn in Reading Journal
Quiz 3
12
Discuss MLA citation
Assignment 7: Issue Research Project
(2 hours) or Fiction Paper (3 hours)
13
Review Wood, Ch. 9
Oral presentations
Turn in Reading Journal
Assessment 2: Oral Presentations Rubric provided
14
Argument Workshop
Assignment 8: Documented Inquiry
(5–6 hours)
15
Final Exam
Sample Class Policies and Expectations
Students often enter college unaware of expectations regarding attendance, participation,
behavior, grading, and academic integrity. Faculty frequently include detailed policies and
expectations in their syllabi, making explicit to students the standards of successful
participation. There are often institutional-level policies as well.
Attendance Policy
Regular attendance is required. School policy states that a student who is absent more
than six hours of class may be viewed as administratively dropped from the course.
Students who intend to withdraw from the course must do so by the official date or they
will receive an “F” in the course. Please be aware that the state of Texas has begun to
impose penalties on students who drop courses excessively. For example, if a student
repeats the same course more than twice, that person will have to pay extra tuition.
Beginning in Fall 2007, the Texas Legislature passed a law limiting students to no more
than SIX total course withdrawals throughout their educational career in obtaining a
certificate and/or degree.
Consideration for excused absences is at the instructorʼs discretion. Excused absences
include official university activities, documented religious observance, illness with a
physicianʼs note, or a verifiable death in the family.
Expectations
Student Conduct
• Students are expected to attend class regularly and to complete reading and
writing assignments on the day specified.
• No food, drink, or tobacco of any kind is permitted in the classroom.
• Cell phones and pagers must be turned off during class.
• Disruptive behavior will not be tolerated.
Improving Alignment Between Postsecondary and Secondary Education
97
Examples of disruptive behavior include, but are not limited, to talking while the
instructor is lecturing, discussing non-course related issues during class
assignment time, and using profane language.
Academic Behaviors
• Students are expected to have adequate time management skills.
• Students are expected to self-monitor their level of competence in the subject in
order to be aware when improvement is needed.
• Students are expected to seek out resources for improvement in understanding
of the subject, such as study groups and tutoring.
• Students are expected to practice efficient study and preparation skills outside of
class.
• Students are expected to take clear, concise class notes.
• Students are expected to be able to collaborate and work in a team.
• Students are expected to be able accept critical feedback including critiques of
submitted work.
•
Grading Policy
Failure to turn in any essay will result in failure of this course!
Grading Scale
• A (90–100) = excellent/performance beyond mastery
• B (80–89) = above average/beyond basic mastery
• C (70–79) = average mastery
• D (60–69) = below average
• F (0–59) = failure
Evaluation Method
• Essay #1 – 5%
• Essay #2 – 10%
• Essay #3 – 15%
• Essay #4 (Research essay) – 25%
• Oral Presentation of research – 5%
• Final Exam – 20%
• Daily Work – 10%
• Quizzes – 10%
Improving Alignment Between Postsecondary and Secondary Education
98
Sample of a grading rubric for essay assignments:
Grading Sheet: Total points possible = 100
1. Does the title tell the reader what the essay is about, and does the essay open
with sufficient background information that sets a context for the topic and
explains the controversy surrounding the topic? (0–10 points)
2. Is the thesis clearly stated? (0–3 points)
3. Do the examples and details in the essay clearly and sufficiently support the
thesis? Does each paragraph relate to the thesis, and does each paragraph have
a clear focus or topic sentence? (0–15 points)
4. Does the writer acknowledge and refute opposing viewpoints? (0–10 pts)
5. Is the essay organized in an easy-to-follow and coherent fashion? (0–5 points)
6. Does the writer provide transition words when needed between sentences, and
does the writer provide transitions between paragraphs? (0–4 points)
7. Is the wording clear, effective, and error free? Does the writer vary sentence
structure, incorporate “vigorous” (active) verbs, and employ concrete, vivid
language that evokes sensory images? Does the writer use a consistent point of
view and verb tense?
(0–10 points)
8. Does the writer wrap up the essay with a concluding paragraph that effectively
ties together the main points of the essay? (0–3 points)
9. Does the writer use proper spelling, grammar, and punctuation? (0–40 points)
Total Points
Academic Integrity Policy
Plagiarism will not be tolerated; be advised we use plagiarism detection software. The
following, directly from the College Handbook, defines and sets the penalties for
plagiarism and/or cheating: All academic work, written or otherwise, submitted by a
student to an instructor or other academic supervisor, is expected to be the result of the
studentʼs own thought, research, or self-expression. In cases in which students feel
uncertain about a question of plagiarism, they are obliged to consult the instructor on
the matter prior to submitting any work. When students submit work purported to be
their own, but which in any way borrows ideas, organization, wording, or anything else
from another source without appropriate acknowledgment, they are guilty of plagiarism.
Plagiarism includes reproducing someone elseʼs work, whether it be from a published
article, chapter of a book, or a paper from a friend.
Plagiarism also includes the practice of employing or allowing another person to alter or
revise the work submitted as oneʼs own. Students must acknowledge all outside
sources of information. Making simple changes while leaving intact the organization and
thoughts of others is plagiaristic. Plagiarism does not, however, extend to those ideas
that are so generally and freely circulated as to be part of the public domain. Cheating
includes, but is not limited to, students wrongfully giving, taking, or presenting any
information or material with the intent of aiding themselves or others with any academic
Improving Alignment Between Postsecondary and Secondary Education
99
work. Within seven working days of finding a student guilty of cheating or plagiarism, an
instructor may choose to do one or more of the following:
• Assign any grade to the paper or test involved and so notify the student in writing.
• Assign any grade for the course in which the offending paper or examination was
submitted and so notify the student in writing.
• Recommend that the student be dropped from the course in which the offense
occurred. Such a recommendation shall be made to the Division Chair.
• Chair shall so recommend to the Dean of Instructional Services and send copies
to the instructor and the student.
• Recommend that the student be dropped from a program of a division. Such a
recommendation shall be made to the Division Chair and to the Division Chair of
the studentʼs program (if different); if agreement occurs, the Division Chair shall
so recommend to the Dean of Instructional Services and send copies to the
instructor and the student.
• Recommend suspension or dismissal from the College. Such a recommendation
shall be made to the Division Chair; if agreement occurs, the Division Chair shall
so recommend to the Dean division and/or the chairperson of the division in
which the student is enrolled in a program, to the instructor, and to the student. If
the Dean concurs on a recommendation of suspension or expulsion from the
College, the Dean shall take the necessary action.
Sample List of Student Resources
The following list is representative of the resources mentioned in 2008 entry-level college
course syllabi in Texas. The resources listed are indicative of the expectation that postsecondary students take responsibility for their own learning. As often explicitly stated in
course syllabi, students are expected to take advantage of these resources, and may even
be required to do so.
•
Writing Center
The English department operates a free-of-charge Writing Center in building X.
Students in English courses can receive in-depth help with writing problems from
the Writing Center staff. Students who believe that they could benefit from
concentrated, supervised work on some aspect of writing should attend the
Writing Center more than once while working on their papers.
•
Student Success Programs
The College supports a variety of programs to help connect students with the
University and foster academic success. These programs include learning
assistance, developmental education, advising and mentoring, admissions and
transition, and federally-funded programs. Students requiring assistance
academically, personally, or socially should contact the Office of Student
Success Programs at xxx-xxx-xxxx for more information and appropriate
referrals.
Improving Alignment Between Postsecondary and Secondary Education
100
•
Americans with Disabilities Act
The college is on record as being committed to both the spirit and letter of
federal equal opportunity legislation; reference Public Law 92-112 of The
Rehabilitation Act of 1973 as amended. With the passage of federal legislation
entitled Americans with Disabilities Act (ADA), pursuant to section 504 of the
Rehabilitation Act, there is renewed focus on providing this population with the
same opportunities enjoyed by all citizens. As a faculty member, instructors are
required by law to provide "reasonable accommodations" to students with
disabilities, so as not to discriminate on the basis of that disability. Student
responsibility primarily rests with informing faculty of their need for
accommodation and in providing authorized documentation through designated
administrative channels. Information regarding specific diagnostic criteria and
policies for obtaining academic accommodations can be found at
www.abc.edu/disability. Also, students may visit the Office for Students with
Disabilities in room X of building X or call (xxx) xxx-xxxx.
•
E-Culture Policy
This course has adopted the College email address as an official means of
communication with students. Through the use of email, the instructor is able to
provide students with relevant and timely information, designed to facilitate
student success. In particular, important information concerning registration,
financial aid, payment of bills, and graduation may be sent to students through
email. All students are assigned an email account and information about
activating and using it is available at www.abc.edu. New students (first semester
at the college) are able to activate their email account 24 hours after registering
for courses. There is no additional charge to students for using this account, and
it remains active as long as a student is enrolled at the college. Students are
responsible for checking their email regularly.
Supplementary Documents
The following list represents additional documents included within the Reference Course
Profile, but not included in the Sample Schedule. These supplementary materials are
intended to enhance both instruction and student learning, and may be used at instructors’
discretion.
•
•
•
•
•
•
Revisions Handout
Using Quotations in a Literary Essay Handout
Works Cited Handout
Explanation of MLA Documentation Style Handout
Peer Response Questions
Bibliography Checklist
Improving Alignment Between Postsecondary and Secondary Education
101
•
•
•
Computer Lab Agenda
Direct Quotes Explanation
Scoring Rubric
Improving Alignment Between Postsecondary and Secondary Education
102
Appendix D: Scale Items & Rationale Statements
For Alignment Between the CCRS and Entry-Level
CTE Courses
Most Necessary for Preparation to Succeed in this Course
• This element is critical for success in the course
• Course is taught with the assumption that students already know this information
• This element will not be retaught in this course
• Students will have difficulty succeeding in the course if they have not learned this
element previously
More Necessary for Preparation to Succeed in this Course
• This element is important for success in the course
• Course is taught with the assumption that students are at least familiar with or
aware of this element
• This element will be reviewed only and not retaught in this course
• Students will benefit from having learned this element previous to the course but
can probably relearn it during the course and still succeed in the course
Less Necessary for Preparation to Succeed in this Course
• Student knowledge of and familiarity with this element may be helpful
• Course is taught with the assumption that students are familiar with this element
on a very general level
• This element will be taught in some detail in the class
• Even if students have not learned this previously, they will be able to learn it in
the course at a level sufficient to succeed in the course
Least Necessary for Preparation to Succeed in this Course
• Students need only minimal knowledge of and familiarity with this element
• Course is taught with the assumption that students may be only vaguely aware of
this element
• This element will be taught as new material in this course even if students have
been taught it before
• Students will be able to succeed in this course even if they only have a very
general awareness or understanding of this element when they enter the course
Not Necessary for Preparation to Succeed in this Course
• This element is too advanced for this course
• This element will be encountered for the first time in subsequent courses in the
subject area
• This element is too specialized or specific for this course
• This element is irrelevant to this course
Improving Alignment Between Postsecondary and Secondary Education
103
•
This element will be introduced as new material in this course with the
assumption that students have not learned anything about it before this course
Improving Alignment Between Postsecondary and Secondary Education
104