| MAJOR FIELD TESTING |
| Planning & Information Book |
| [major field testing main menu] [planning & information book in printable format pdf] |
Background
Contact InformationPART I: The Planning Year
Major Field Testing Q&A, part 1
Types of Tests:
Standardized
Locally-DevelopedPART II: The Testing Year
Major Field Testing Q&A, part 2
Test Security
Students Exemptions
The Reporting ChartAPPENDICES
Appendix A: Performance Standard for Major Field Testing
Appendix B: Financial Statement
Appendix C: Sample Letters & Correspondence
The 2005-2010 guidelines for the Performance Funding Standards state:
Since 1967, the Tennessee Higher Education Commission (THEC) has been responsible for promoting, coordinating, and unifying Tennessee higher education. Overall, THEC is responsible for a variety of higher education statutory requirements that range from funding formula issues to strategic planning to academic program approval. Since the beginning of the performance funding program in 1978, [THEC] has coordinated the performance funding initiative, an incentive-based funding initiative for public higher education that financially rewards exemplary institutional performance on selected measures of effectiveness. The program is administered on five-year cycles…. Through the performance funding program, THEC continues to promote its traditional principles of equity, excellence, accessibility, and accountability. The main emphasis of the program is a fundamental commitment to promote improvement in quality and performance of public colleges and universities.
Major field assessment is designed to provide incentives for institutions to improve the quality of major field programs as evaluated by the performance of graduates on approved examinations. During each five year cycle, UT must provide a set of test scores in major fields for every graduate in those undergraduate fields that average ten or more graduates per year. Each affected department must report scores to THEC once during that five year cycle. New during the 2005-2006 performance funding cycle, all licensed areas (specifically Education, Engineering, and Nursing) must submit their scores annually.
This requirement is important. UT can earn up to 5.45% of its annual budget based on how well students perform on major field tests and other standards of performance.
MICHAEL
W. McFALL II
Assessment Coordinator
THE UNIVERSITY
OF TENNESSEE
Office of Institutional Research and Assessment
2538 Dunford Hall · 915 Volunteer Blvd
600 Henley Street
Knoxville TN 37996-4122
ph: 865.974.4373
fax: 865.974.1428
email: mmcfall@tennessee.edu
Major Field Testing Q & A, part 1
Who is tested?
All students graduating in both the fall and spring semesters of the testing
and reporting year must be tested. While THEC would prefer that all students
graduating in a year (which, for them, runs as Summer/Fall/Spring) be
tested, testing of summer graduates is not required.
What types of tests can be used?
There are several options that can be used. Many departments opt for a
locally developed instrument while others use a capstone course or culminating
experience as their major field assessment. Another option is to employ
one of the many standardized tools available on the market. The choice
of which assessment method to use is entirely up to each department (see
Types of Tests).
When should tests be given?
This is often dictated by the type of test that is used. Obviously a locally
developed instrument can be administered at the discretion of the department
while many licensure-type tests are controlled by central agencies. On
the whole, tests can be offered at times that are convenient for both
faculty and students provided that all eligible students (fall and spring
graduates) are tested.
What is the general calendar or timeline for
the Planning Year?
For those using a standardized test, the objective for the Planning Year
is to confirm the choice of instruments. For nationally scheduled tests
it might also be necessary to confirm available test dates.
For those using a local test there are other factors to consider. If a department is using an established test, then the Planning Year should be used to make minor revisions, ensuring that the instrument is ready for administration the following year. If a new test is being developed, then the Planning Year is used for test construction, external review, and baseline administration.
Is baseline testing mandatory?
Baseline testing is required when a department develops a new local test.
For departments making revisions of less than 20% on an old test or choosing
to begin using a new nationally-normed standardized test, baseline testing
is optional.
What is the benefit of major field testing?
Overall, major field testing accounts for 10% of the performance funding
allotment. For each area that tests, a one-time, flat bonus as well as
a performance-based bonus is appropriated (see Appendix
B: Financial Statement).
There are several tools that a department can use when choosing a test for major field assessment. Basically it comes down to what works best for each department. Below are some examples of the types of tests that can be used.
The
Standardized Test
Many majors have utilized a variety of available standardized instruments.
Often these tests can serve a dual purpose. For example, Nursing and Architecture
both use a national licensing exam as their major field assessment test.
Additionally, Education and Engineering use a national exam for their
assessment. These national tests are centrally scheduled, and it is the
department’s responsibility to inform students of registration dates.
For other areas, specialty tests are available through organizations such as Educational Testing Service. Given that they can be ordered and administered locally, such instruments offer departments the advantages of minimal time commitment with regard to development and planning in addition to greater flexibility in scheduling.
While there are few details to consider when using a standardized test,
there are still important factors that should be addressed. Below is a
general timeline for those using a standardized instrument.
| Timeline |
Action |
|
Using
a previously used test |
Using
a test for the first time |
|
| During the Planning Year... |
|
|
| Summer/Fall Semesters |
|
|
| Spring Semester |
|
|
| During the Testing Year... |
|
|
Another option available to departments, and one that many prefer, is
to develop their own assessment tools. There are both advantages and disadvantages
to using a locally developed test. Using such a test allows for the major
field assessment instrument to relate directly to current curricula. However,
departments must invest time into maintaining such an instrument between
test administrations. In addition, there are considerable time commitments
and planning that must occur if a department is developing a new local
test.
Similar to a locally developed test, departments may choose to use a capstone course or a culminating project as an assessment of the major. Though this enables a department to use a pre-established measure, it still requires a time commitment in development and maintenance.
If a department chooses to use a locally developed instrument to assess the major, there are some guidelines that must be followed. If an existing test is to be reused, departments are permitted to alter approximately 20% of the test without penalty. In fact, such revisions are encouraged and recommended. This small change ensures continuity and comparability of results while allowing for changes in curriculum that may occur between test administrations.
If the changes to an existing test exceed 20% or if a department has decided to develop a new local test, there are a few more constraints, rules and deadlines that must be followed:
1. First
and foremost, planning for a new locally developed test should ideally
begin in the summer or early fall semester of the Planning Year (or one
year prior to the test administration year).
2. During the planning phase, a proposal, including methodology, should
be written.
3. While the test is being developed, the department should begin securing
external consultants to review the test. THEC requires that all new, locally
developed tests are reviewed by either two specialists in the discipline
(both of whom must be outside the state of Tennessee) or by one discipline
specialist (outside the state) and one measurement consultant (this individual
can be from inside the state or even from UT).
4. After the reviews have taken place and all updates/changes have been
made, the department should forward copies of the following to OIRA for
filing purposes as well as for forwarding on to THEC:
a. A copy of each consultants’ vita,
b. Copies of all correspondence to and from the consultants relating to
the review,
c. Copies of both the test proposal and the test itself.
5. Sometime in the spring semester of the Planning Year, the department
must baseline the test. This will provide a basis for comparison during
the actual administration of the test.
6. Following the baseline administration, the test scores should also
be forwarded to OIRA.
Below is a general timeline overview for those using a locally developed instrument.
Timeline |
Action |
|
Developing
a new test |
Reusing
an existing test |
|
| During the Planning Year... | ||
| Summer/Fall Semesters |
|
|
|
|
|
|
||
|
||
| Spring Semester |
|
|
|
||
|
||
| During the Testing Year... |
|
|
|
||
Major Field Testing Q & A, part 2
Who is eligible to be tested?
In the middle of each term, OIRA will distribute a list of all students
who have completed their applications for graduation. It is the department’s
responsibility to make sure that everyone on that list is accounted for.
In addition, departments should make general announcements to potential
graduating seniors about this testing requirement in order to ensure that
the list is as accurate as possible. Remember, all seniors graduating
from departments undergoing major field testing must participate.
How are students notified that they need to
take the test?
Notification of students is the responsibility of the department. Using
the list of seniors who applied to graduate, proactive communication using
multiple mediums, such as letters, class announcements, and e-mails, is
the best strategy for student notification. (Appendix
C contains some sample notification letters.)
Can students be exempted from major field
testing?
As stated earlier, every student graduating from a program undergoing
major field testing must test. However, there are exceptions to every
rule. Occasionally there are circumstances that would allow a student
to be exempted from major field testing. It is up to the department to
determine if a student should be granted an exemption. However, exemptions
should only be granted for extenuating circumstances, not for simple inconvenience.
(See Student Exemptions for a list of exemption
explanations.)
As a department, what can be done to maximize
performance?
Discussion sometimes occurs regarding whether departments should help
students prepare for taking a major field test. There are at least as
many differing opinions of this as there are colleges! Some areas using
national exams that are also required for licensure have held study sessions.
Other areas have held informal Q&A sessions to help explain the test
and the rationale for using it. Whatever the decision, common sense and
good judgment should prevail.
What can be done to encourage participation?
One of the easiest ways to encourage participation is, if possible, to
schedule multiple test dates at various times. This will allow for variability
in students’ schedules. Many departments have offered additional
incentives to students, some based on mere participation and others on
actual performance. For example, in those areas where a capstone project
or experience is used, students receive actual course credit. Other areas
have offered refreshments or snacks during or after the exam. Still others
have incorporated a prize system in which the students who earn the highest
scores receive gift certificates. These incentive programs are at the
discretion of the department. Finally, the best way to maximize participation
is through good communication, both early and often.
Departments are responsible for maintaining all tests in a secure location. If there are any questions regarding procedures for administration of a local or standardized test or details relating to security and confidentiality, please call OIRA. With standardized tests from organizations like ETS and NTE, strictly regulated measures are required. Please be certain, for example, that tests are stored in a locked storage area, carefully inventoried, and administered according to the usually detailed directions provided.
Following each test date, if OIRA ordered standardized tests from the
testing company, then the standardized test booklets, answer sheets, and
related materials must be returned directly to us. We will assume the
responsibility for final return mailing of materials to the test company,
but departments are responsible for retaining all used and unused materials
and for completing an inventory form upon receipt and return of all materials
to us.
Student Exemptions
It is the testing department's responsibility to test all of their graduating
seniors. However, it is sometimes necessary to exempt a student based
on different circumstances. As a general rule of thumb, the number of
exemptions granted should be no greater than 5% of the total number of
students undergoing major field testing. THEC requires OIRA to send a
list of exempted students with an explanation for each exemption for approval
to the chief academic officer. It should be noted that the department—not
OIRA— determines if a student should be given an exemption. Below
is a list of exemption codes to be used when granting student exemptions.
EXEMPTION CODES
1. The
student is graduating by correspondence; returning to take the examination
would be a hardship.
2. The student completed most of his or her course work more than four
years ago; the test score would not reflect the effectiveness of current
curriculum.
3. The department either did not know the student was a major or thought
the student had already graduated.
4. According to the department's records, the student is not a declared
major.
5. According to the department's records, the student has not actually
graduated and will not graduate this semester.
6. There was some other circumstance which the department believes is
appropriate for granting an exemption. Please send an individual explanation
when requesting an exemption that cannot be categorized under any of the
five previous codes.
The
Major Field Reporting Chart
As mentioned earlier, each testing area will receive a list of students
who have applied to graduate. In addition to student names, this list
will also include columns for tracking test completion, exemption codes,
and test scores. Collectively, this list is called the Major Field Reporting
Chart. This chart is intended to make the process of tracking students
and reporting outcomes much simpler. Listed below are more in depth descriptions
and instructions for completing the chart.
DESCRIPTION AND INSTRUCTIONS
After receiving the Major Field Reporting Chart…
In
General
Review the names on the chart. Add names as necessary and scratch out
those who you know will NOT be graduating this term.
The
Taken and Score Columns
After a student has taken the test, make a mark in the TAKEN column and,
if scoring is done in-house, fill in the score for each student.
The
Exempt Column
If an exemption is to be granted, list the reason code for that student
(see Student Exemptions for a list of possible reasons)
Appendix
A: Performance Standard for Major Field Testing.
Points: 10 points
Purpose:
This indicator is designed to provide incentives for institutions
to improve the quality of major field programs as evaluated by the performance
of graduates on approved examinations.
Evaluation: A major field will be considered successful if the test score
is either at or above a recognized norm or shows improvement over the
institution's most recent test score (or a baseline score for a new test).
All programs will be reported once during the five-year cycle with the
exception of licensure programs. All licensure programs will be reported
annually.
Processes:
1. Prior to the beginning of the cycle, a list of approved major field
tests and measures will be developed by the Commission. During the cycle,
tests may be submitted through the governing boards to the Commission
for consideration for inclusion to the approved list.
2. In major areas in which national standardized tests are not available,
or where faculty do not consider available tests appropriate, institutions
may develop test instruments - either on a single campus or in concert
with other institutions. If such tests are developed, plans should be
made for pilot testing to provide for evaluation and to develop scores
for subsequent comparison for scoring purposes. The plan for test construction
must include a schedule of activities, sampling procedures, and credentials
of cooperating institutional staff or credentials of external consultants.
These plans should be submitted to both the governing boards and Commission
staff for prior approval.
3. If an institution develops a local test instrument, no more than 20%
of the test content may be modified between testing cycles. This ensures
data continuity and comparability of results. A locally developed test
that is changed ny 20% or more will be treated as a new test and must
adhere to the same procedure as stated above.
4. Institutions must submit a test schedule which ensures that approximately
20% of programs are tested each year. This schedule must be approved by
the appropriate governing board and Commission staff. Each institution
will notify the governing board and Commission of its test schedule for
all programs by October 15, 2005.
5. All licensure programs at the associate and baccalaureate level will
be reported annually. Licensure programs include engineering, health and
teach education.
6. Scoring will be cumulative and new scores will be added in each succeeding
year of the cycle.
7. In programs for which national norms are not appropriate, but for which
pass rates are appropriate, comparisons may be made to those pass rates
and prior institutional pass rates.
8. When a program is assessed for this standard, all students graduating
in that program must be tested. Exceptions for individual students (for
good cause) must be approved by the chief academic officer. Exceptions
should not be approved for simple inconvenience.
9. For purposes of this standard, a major field is defined as all programming
at one degree level bearing the same name. For example, a B.A. and B.S.
in Psychology would be considered as one field. Other closely related
fields may be considered as one field at the request of the institution
and the approval of the governing board and the Commission.
10. If both associate and baccalaureate degrees are offered in a field
and if testing is appropriate to both levels (e.g., nursing), then all
graduates at both levels must be tested and reported.
11. Programs will be exempt from the requirements of this indicator if
any of the following conditions exist:
• Program is a certificate program.
• Program
is a performance-oriented program in the fine or performing arts.
• Program is interdisciplinary, multidisciplinary, or self-designed
to include several related fields;
• Baccalaureate programs that have not generated an average of 10
students per year, or a minimum of 50 graduates during the time period
of 1999-00 to 2003-04. Associate programs that have not generated an average
of 10 students per year or a minimum of 30 greaduates during the time
period of 2001-02 to 2003-04.
• Program is in phase-out or inactive status at the beginning of
the cycle. If a program becomes inactive during the cycle, the scheduled
program shall be exempt from the major field testing requirement.
• New programs approved during the 2000-05 performance funding cycle
that have not reached maturity will be exempt from the testing requirement.
program maturity for associate degree programs is a three year period
after implementation; program maturity for baccalaureate programs is a
five year period after implementation. New associate degree programs approved
by yhr Commission after the 2005-10 cycle begins unless they are excluded
due to one of the other exemptions and reached program maturity during
the cycle must be scheduled for testing. Since baccalaureate programs
require five years to mature, new baccalaureate programs that were approved
after July 2005, would not mature until 2011 and thus would be exempt
from the major field requirement.
12. Institutions may submit other programs for exemption through their
respective governing board for consideration by the Commission.
13. A copy of the results from the scoring template must be included with
the Performance Funding Reporting Template for each reported program.
Scoring:
Performance for major field assessment will be evaluated by comparing
the program's average score with an external norm or institution's most
recent test score. Each major field test should be compared to national
norms or national pass rates when available. if national data are not
available, then a comparison is made to the most recent test score (or
a baseline score for new tests).
This comparison
is made by dividing the institutional average by its comparison score
average for that cycle year (no percent attainment may exceed 100%). This
overall percentage will be rounded to the nearest whole percentage which
will be compared with Table 2 to award points for the Major Field Assessment
standard.
Table 2
%
Insitution to Nat'l Avg/Comp Score |
Below 70% |
70% to 74% | 75% to 78% | 79% to 81% | 82% to 84% | 85% to 87% | 88% to 90% | 91% to 93% | 94% to 96% | 97% to 99% | 100% |
Points |
0 |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
Appendix B: Financial Statement
DURING THE PLANNING YEAR…
1. Consultants
for a new locally developed test: $500 Total
When departments devise a locally developed test, two out-of-state consultants
review the test. Two specialists in the discipline, or one in the discipline
and one who is a measurement consultant or specialist, are required.
2. Consultants
for a local test revision: Up to $200
Departments may elect to obtain advice as they revise a test (under 20%
of the test). Hiring a consultant is not a requirement, but OIRA will
support this cost when requested.
3. Pilot
testing for new standardized tests: Varies
In the planning year, when a department chooses to use a new test in this
cycle, funding is available for standardized test costs necessary to establish
an institutional baseline against which to compare the reporting year
score. Departments consult with OIRA to establish the number of test scores
needed to set this baseline and the costs which can be covered.
DURING THE TESTING YEAR…
1. Funding
for actual test costs: Varies
A. Registration fees
If a test requires students to pay a registration fee up front (e.g.,
FE for Engineering, PRAXIS), OIRA will fully reimburse those students
for the test fees. (EXCEPTION: In instances where all graduating or recently
graduated students are annually required to take a test in order to receive
licensure – such as in Nursing and Architecture – no reimbursement
is available for the students’ costs in taking these tests.)
B. Purchase
of standardized tests
Departments using standardized tests they administer within the department
(e.g., ETS’s MFAT) may purchase these tests on their own or request
OIRA to arrange the ordering of these tests. In either case, OIRA will
cover the costs of the test.
C. Miscellaneous
costs associated with local tests
There are many “hidden” costs that can be associated with
a locally developed test. These can include making copies of the test
itself, purchasing scan sheets, and scoring scan sheets. OIRA is available
to help with any and all of these types of costs.
FOLLOWING THE TESTING YEAR…
1. Chancellor's
Bonus: $1,000
This flat fee is paid by the Chancellor’s Office at the successful
completion of the testing/reporting year. When two majors or concentrations
are housed in a single department, the $1,000 bonus is divided between
the majors. This occurs whether or not the same or different tests are
used to meet this requirement.
Special Note for Licensure Areas: Due to the fact that licensure scores
are reported annually, the Chancellor's Bonus will be divided into 5 equal,
annual installments of $200.
2. Performance-Contingent
Bonus: Varies
This fee will be paid to departments based on the following funding formula:
X x Y =
Z, where: X = the number of students testing/reporting
Y = a dollar figure based on the weighting figure from the outcome
Z = the amount paid to the department*
To calculate "Y", use the following:
%
Insitutional Score to Comparison Score |
The
Per Student Bonus Payment (Y in the formula) |
100% |
$30 |
97%
to 99% |
$15 |
Below
97% |
$0 |
Special Note for Licensure Areas: Licensure areas will receive their Performance-Contingent bonus at the end of the five-year performance funding cycle (along with those areas that test in the fifth year of the cycle). The bonus amount will be based on the average number of students and overall performance.
*the total performance-contingent bonus is capped at $4,000.
Appendix C: Sample Letters & Correspondence
Sample Correspondence
This appendix
contains copies of letters departments have sent to graduating seniors
informing them about their major field testing obligation as well as samples
used in other facets of this accountability effort.
If you have any questions concerning the letters in this appendix, or
wish additional samples, call OIRA, 974-4373.
Description
Letter
of Invitation to a Local Test Development Consultant
Sample Letters about Required Testing
Romance
Languages
English
1
English
2
Engineering
1
Engineering
2
Mathematics
1
Mathematics
2
Baseline
Testing Invitation
Thank
You for Helping Establish Our Baseline
Informing
Faculty about the Requirement (in this example, the testing will
take place in the instructor's class)
Survey
Attached to a Major Field Test(illustrating another way to use
the requirement to positive effect)