 |
||
 |
 |
 |
 |
Prepared by The Mathematical Association of America, Subcommittee on Assessment, Committee on the Undergraduate Program in Mathematics
Approved by CUPM at the San Francisco meeting, January 4, 1995
Preface
Recently there has been a series of reports and recommendations about all aspects of the undergraduate mathematics program. In response, both curriculum and instruction are changing amidst increasing dialogue among faculty about what those changes should be. Many of the changes suggested are abrupt breaks with traditional practice; others are variations of what has gone on for many decades. Mathematics faculty need to determine the effectiveness of any change and institutionalize those that show the most promise for improving the quality of the program available to mathematics majors. In deciding which changes hold the greatest promise, student learning assessment provides invaluable information. That assessment can also help departments formulate responses for program review or other assessments mandated by external groups.
The Committee on the Undergraduate Program in Mathematics established the Subcommittee on Assessment in 1990. This document, approved by CUPM in January 1995, arises from requests from departments across the country struggling to find answers to the important new questions in undergraduate mathematics education. This report to the community is suggestive rather than prescriptive. It provides samples of various principles, goals, areas of assessment, and measurement methods and techniques. These samples are intended to seed thoughtful discussions and should not be considered as recommended for adoption in a particular program, certainly not in totality and not exclusively.
Departments anticipating program review or preparing to launch the assessment cycle described in this report should pay careful attention to the MAA Guidelines for Programs and Departments in Undergraduate Mathematical Sciences [1]. In particular, Section B.2 of that report and step 1 of the assessment cycle described in this document emphasize the need for departments to have
a. A clearly defined statement of program mission; and
b. A delineation of the educational goals of the program.
The Committee on the Undergraduate Program in Mathematics urges departments to consider carefully the issues raised in this report. After all, our programs should have clear guidelines about what we expect students to learn and have a mechanism for us to know if in fact that learning is taking place.
James R. C. Leitzel, Chair, Committee on the Under-graduate
Program in Mathematics, 1995
Membership of the
Subcommittee on Assessment, 1995
Larry A. Cammack, Central
Missouri State University, Warrensburg, MO
James
Donaldson, Howard University, Washington, DC
Barbara T.
Faires, Westminster College, New Wilmington, PA
Henry
Frandsen, University of Tennessee, Knoxville, TN
Robert T.
Fray, Furman University, Greenville, SC
Rose C. Hamm,
College of Charleston, Charleston, SC
Gloria C. Hewitt,
University of Montana, Missoula, MT
Bernard L. Madison
(Chair), University of Arkansas, Fayetteville, AR
William
A. Marion, Jr., Valparaiso University, Valparaiso, IN
Michael Murphy, Southern College of Technology, Marietta,
GA
Charles F. Peltier, St. Marys College, South Bend, IN
James W. Stepp, University of Houston, Houston, TX
Richard D. West, United States Military Academy, West
Point, NY
I. Introduction
The most important indicators of effectiveness of mathematics degree programs are what students learn and how well they are able to use that learning. To gauge these indicators, assessment — the process of gathering and interpreting information about student learning — must be implemented. This report seeks to engage faculty directly in the use of assessment of student learning, with the goal of improving undergraduate mathematics programs.
Assessment determines whether what students have learned in a degree program is in accord with program objectives. Mathematics departments must design and implement a cycle of assessment activity that answers the following three questions:
Each step of an ongoing assessment cycle broadens the knowledge of the department in judging the effectiveness of its programs and in preparing mathematics majors. This knowledge can also be used for other purposes. For example, information gleaned from an assessment cycle can be used to respond to demands for greater accountability from state governments, accrediting agencies, and university administrations. It can also be the basis for creating a shared vision of educational goals in mathematics, thereby helping to justify requests for funds and other resources.
This report provides samples of various principles, goals, areas of assessment, and measurement methods and techniques. Many of the items in these lists are extracted from actual assessment documents at various institutions or from reports of professional organizations. These samples are intended to stimulate thoughtful discussion and should not be considered as recommended for adoption in a particular program, certainly not in totality and not exclusively. Local considerations should guide selection from these samples as well as from others not listed.
II. Guiding Principles
An essential prerequisite to constructing an assessment cycle is agreement on a set of basic principles that will guide the process, both operationally and ethically. These principles should anticipate possible problems as well as ensure sound and effective educational practices. Principles and standards from several sources (see references 2,3,4,5,and 6) were considered in the preparation of this document, yielding the following for consideration:
a. Objectives should be realistically matched to institutional goals as well as to student backgrounds, abilities, aspirations, and professional needs.
b. The major focus of assessment (by mathematics departments) should be the mathematics curriculum.
c. Assessment should be an integral part of the academic program and of program review.
d. Assessment should be used to improve teaching and learning for all students, not to filter students out of educational opportunities.
e. Students and faculty should be involved in and informed about the assessment process, from the planning stages throughout implementation.
f. Data should be collected for specific purposes determined in advance, and the results should be reported promptly.
III. The Assessment Cycle
Once the guiding principles are formulated and understood, an assessment cycle can be developed:
1. Articulate the learning goals of the mathematics curriculum and a set of objectives that should lead to the accomplishment of those goals.
2. Design strategies (e.g., curriculum and instructional methods) that will accomplish the objectives, taking into account student learning experiences and diverse learning styles, as well as research results on how students learn.
3. Determine the areas of student activities and accomplishments in which quality will be judged. Select assessment methods designed to measure student progress toward completion of objectives and goals.
4. Gather assessment data; summarize and interpret the results.
5. Use the results of the assessment to improve the mathematics major.
Steps 1 and 2 answer the first question in the introduction — what should the students learn? Steps 3 and 4, which answer the second question about how well they are learning, constitute the assessment. Step 5 answers the third question on what improvements are possible.
Step 1. Set the Learning Goals and Objectives
There are four factors to consider in setting the learning goals of the mathematics major: institutional mission, background of students and faculty, facilities, and degree program goals. Once these are well understood, then the goals and objectives of the major can be established. These goals and objectives of the major must be aligned with the institutional mission and general education goals and take into account the information obtained about students, faculty, and facilities.
Institutional Mission and Goals. The starting point for establishing goals and objectives is the mission statement of the institution. Appropriate learning requirements from a mission statement should be incorporated in the departments goals. For example, if graduates are expected to write with precision, clarity, and organization within their major, this objective will need to be incorporated in the majors goals. Or, if students are expected to gain skills appropriate for jobs, then that must be a goal of the academic program for mathematics majors.
Information on Faculty, Students, and Facilities. Each institution is unique, so each mathematics department should reflect those special features of the institutional environment. Consequently, the nature of the faculty, students, courses, and facilities should be studied in order to understand special opportunities or constraints on the goals of the mathematics major. Questions to be considered include the following:
Goals and Objectives of Mathematics Degree Program. A degree program in mathematics includes general education courses as well as courses in mathematics. General education goals should be articulated and well-understood before the goals and objectives of the mathematics curriculum are formulated. Of course, the general education goals and the mathematics learning goals must be complementary and consistent [6, pages 183-223]. Some examples of general education goals that will affect the goals of the degree program and what learning is assessed include the following:
Graduates are expected to speak and write with precision, clarity, and organization; to acquire basic scientific and technological literacy; and to be able to apply their knowledge.
Degree programs should prepare students for immediate employment, graduate schools, professional schools, or meaningful and enjoyable lives.
Degree programs should be designed for all students with an interest in the major subject and encourage women and minorities, support the study of science, build student self-esteem, ensure a common core of learning, and encourage life-long learning.
Deciding what students should know and be able to do as mathematics majors ideally is approached by setting the learning goals and then designing a curriculum that will achieve those goals. However, since most curricula are already structured and in place, assessment provides an opportunity to review curricula, discern the goals intended, and rethink them. Curricula and goals should be constructed or reviewed in light of recommendations on the mathematics major as contained in the 1991 CUPM report on the Undergraduate Major in the Mathematical Sciences [6, pages 225-247].
Goal setting should move from general to specific, from program goals to course goals to assessment goals. Goals for student learning can be statements of knowledge students should gain, skills they should possess, attitudes they should develop, or requirements of careers for which they are preparing. The logical starting place for discerning goals for an existing curriculum is to examine course syllabi, final examinations, and other student work.
Some samples of learning goals are:
Mathematical Reasoning. Students should be able to perform complex tasks; explore subtlety; discern patterns, coherence, and significance; undertake intellectually demanding mathematical reasoning; and reason rigorously in mathematical arguments.
Personal Potential. Students should be able to undertake independent work, develop new ideas, and discover new mathematics. Students should possess an advanced level of critical sophistication; knowledge and skills needed for further study; personal motivation and enthusiasm for studying and applying mathematics; and attitudes of mind and analytical skills required for efficient use, appreciation, and understanding of mathematics.
Nature of Mathematics. Students should possess an understanding of the breadth of the mathematical sciences and their deep interconnecting principles; substantial knowledge of a discipline that makes significant use of mathematics; understanding of interplay among applications, problem-solving, and theory; understanding and appreciation of connections between different areas of mathematics and with other disciplines; awareness of the abstract nature of theoretical mathematics and the ability to write proofs; awareness of historical and contemporary contexts in which mathematics is practiced; understanding of the fundamental dichotomy of mathematics as an object of study and a tool for application; and critical perspectives on inherent limitations of the discipline.
Mathematical Modeling. Students should be able to apply mathematics to a broad spectrum of complex problems and issues; formulate and solve problems; undertake some real-world mathematical modeling project; solve multi-step problems; recognize and express mathematical ideas imbedded in other contexts; use the computer for simulation and visualization of mathematical ideas and processes; and use the process by which mathematical and scientific facts and principles are applied to serve society.
Communication and Resourcefulness. Students should be able to read, write, listen, and speak mathematically; read and understand technically-based materials; contribute effectively to group efforts; communicate mathematics clearly in ways appropriate to career goals; conduct research and make oral and written presentations on various topics; locate, analyze, synthesize, and evaluate information; create and document algorithms; think creatively at a level commensurate with career goals; and make effective use of the library. Students should possess skill in expository mathematical writing, have a disposition for questioning, and be aware of the ethical issues in mathematics.
Content Specific Goals. Students should understand theory and applications of calculus and the basic techniques of discrete mathematics and abstract algebra. Students should be able to write computer programs in a high level language using appropriate data structures (or to use appropriate software) to solve mathematical problems.
Topic or thematic threads through the curriculum are valuable in articulating measurable objectives for achieving goals. Threads also give the curriculum direction and unity, with courses having common purposes and reinforcing one another. Each course or activity can be assessed in relation to the progress achieved along the threads. Possible threads or themes are numerous and varied, even for the mathematics major. Examples include mathematical reasoning, communication, scientific computing, mathematical modeling, and the nature of mathematics. The example of a learning goal and instructional strategy in the next section gives an idea of how the thread of mathematical reasoning could wind through the undergraduate curriculum.
Step 2. Design Strategies to Accomplish Objectives
Whether constructing a curriculum for predetermined learning goals or discerning goals from an existing curriculum, strategies for accomplishing each learning goal should be designed and identified in the curricular and co-curricular activities. Strategies should respect diverse learning styles while maintaining uniform expectations for all students.
Strategies should allow for measuring progress over time. For each goal, questions such as the following should be considered.
The following example of a goal and strategy can be made more specific by referencing specific courses and activities in a degree program.
Learning goal. Students who have completed a mathematics major should be able to read and understand mathematical statements, make and test conjectures, and be able to construct and write proofs for mathematical assertions using a variety of methods, including direct and indirect deductive proofs, construction of counterexamples, and proofs by mathematical induction. Students should also be able to read arguments as complex as those found in the standard mathematical literature and judge their validity.
Strategy. Students in first year mathematics courses will encounter statements identified as theorems which have logical justifications provided by the instructors. Students will verify the need for some of the hypotheses by finding counterexamples for the alternative statements. Students will use the mathematical vocabulary found in their courses in writing about the mathematics they are learning. In the second and third years, students will learn the fundamental logic needed for deductive reasoning and will construct proofs of some elementary theorems using quantifiers, indirect and direct proofs, or mathematical induction as part of the standard homework and examination work in courses. Students will construct proofs for elementary statements, present them in both written and oral form, and have them critiqued by a mathematician. During the third and fourth years, students will formulate conjectures of their own, state them in clear mathematical form, find methods which will prove or disprove the conjectures, and present those arguments in both written and oral form to audiences of their peers and teachers. Students will make rational critiques of the mathematical work of others, including teachers and peers. Students will read some mathematical literature and be able to rewrite, expand upon, and explain the proofs.
Step 3. Determine Areas and Methods of Assessment
Learning goals and strategies should determine the areas of student accomplishments and departmental effectiveness that will be documented in the assessment cycle. These areas should be as broad as can be managed, and may include curriculum (core and major), instructional process, co-curricular activities, retention within major or within institution, and success after graduation. Other areas such as advising and campus environment may be areas in which data on student learning can be gathered.
Responsibility for each chosen area of assessment should be clearly assigned. For example, the mathematics faculty should have responsibility for assessing learning in the mathematics major, and the college may have responsibility for assessment in the core curriculum.
Assessment methods should reflect the type of learning to be measured. For example, the Graduate Record Examination (GRE) may be appropriate for measuring prepara tion for graduate school. On the other hand, an attitude survey is an appropriate tool for measuring an aptitude for life-long learning. An objective paper-and-pencil examination may be selected for gauging specific content knowledge.
Eight types of assessment methods are listed below, with indications of how they can be used. Departments will typically use a combination of methods, selected in view of local program needs.
1. Tests. Tests can be objective or subjective, multiple-choice or free-response. They can be written or oral. They can be national and standardized, such as the GRE and Educational Testing Service Major Field Achievement Test, or they can be locally generated. Tests are most effective in measuring specific knowledge and its basic meaning and use.
2. Surveys. These can be written or they can be compiled through interviews. Groups that can be surveyed are students, faculty, employers, and alumni. Students can be surveyed in courses (about the courses), as they graduate (about the major), or as they change majors (about their reasons for changing).
3. Evaluation reports. These are reports in which an individual or group is evaluated through a checklist of skills and abilities. These can be completed by faculty members, peers, or employers of recent graduates. In some cases, self-evaluations may be used, but these tend to be of less value than more objective evaluations. Grades in courses are, of course, fundamental evaluation reports.
4. Portfolios. Portfolios are collections of student work, usually compiled for individual students under faculty supervision following a standard departmental protocol. The contents may be sorted into categories, e.g., freshman or sophomore, and by type, such as homework, formal written papers, or examinations. The work collected in a student's portfolio should reflect the student's progress through the major. Examples of work for portfolios include homework, examination papers, writing samples, independent project reports, and background information on the student. In order to determine what should go in a portfolio, one should review what aspects of the curriculum were intended to contribute to the objectives and what work shows progress along the threads of the curriculum. Students may be given the option of choosing what samples of particular types of work are included in the portfolio.
5. Essays. Essays can reveal writing skills in mathematics as well as knowledge of the subject matter. For example, a student might write an essay on problem-solving techniques. Essays should contribute to learning. For example, students might be required to read four selected articles on mathematics and, following the models of faculty-written summaries of two of them, write summaries of the other two. Essays can be a part of courses and should be candidates for inclusion in portfolios.
6. Summary courses. Such courses are designed to cover and connect ideas from across the mathematics major. These may be specifically designed as summary courses and as such are usually called capstone courses, or they may be less specific, such as senior seminars or research seminars. Assessment of students performances in these courses provides good summary information about learning in the major.
7. Oral presentations. Oral presentations demonstrate speaking ability, confidence, and knowledge of subject matter. Students might be asked to prepare an oral presentation on a mathematics article. If these presentations are made in a summary course setting, then the discussion by the other students can serve both learning and assessment.
8. Dialogue with students. Student attitudes, expectations, and opinions can be sampled in a variety of ways and can be valuable in assessing learning. Some of the ways are student evaluations of courses, interviews by faculty members or administrators, advising interactions, seminars, student journals, and informal interactions. Also, in-depth interviews of individual students who have participated in academic projects as part of a group can provide insights into learning from the activities.
Student cooperation and involvement are essential to most assessment methods. When selecting methods appropriate to measuring student learning, faculty should exercise care so that all students are provided varied opportunities to show what they know and are able to do. The methods used should allow for alternative ways of presentation and response so that the diverse needs of all students are taken into account, while ensuring that uniform standards are supported. Students need to be aware of the goals and methods of the departmental assessment plan, the goals and objectives of the mathematics major and of each course in which they enroll, and the reason for each assessment measurement. In particular, if a portfolio of student work is collected, students should know what is going to go into those portfolios and why. Ideally, students should be able to articulate their progress toward meeting goals — in each course and in an exit essay at the end of the major.
Since some assessment measures may not affect the progress of individual students, motivation may be a problem. Some non-evaluative rewards may be necessary.
Step 4. Gather Assessment Data
After the assessment areas and methods are determined, the assessment is carried out and data documenting student learning are gathered. These data should provide answers to the second question in the introduction — how well are the students learning?
Careful record keeping is absolutely essential and should be well-planned, attempting to anticipate the future needs of assessment. Additional record storage space may be needed as well as use of a dedicated computer database. The data need to be evaluated relative to the learning goals and objectives. Evaluation of diverse data such as that in a student portfolio may not be easy and will require some inventiveness. Standards and criteria for evaluating data should be set and modified as better information becomes available, including longitudinal data gathered through tracking of majors through the degree program and after graduation. Furthermore, tracking records can provide a base for longitudinal comparison of information gathered in each pass through the assessment cycle.
Consistency in interpreting data, especially over periods of time, may be facilitated by assigning responsibility to a core group of departmental faculty members.
Ways to evaluate data include comparisons with goals and objectives and with preset benchmarks; comparisons over time; comparisons to national or regional norms; comparisons to faculty, student, and employer expectations; comparisons to data at similar institutions; and comparisons to data from other majors within the same institution.
If possible, students should be tracked from the time they apply for admission to long after graduation. Their interests at the time of application, their high school records, their personal expectations of the college years, their curricular and extracurricular records while in college, their advanced degrees, their employment, and their attitudes toward the institution and major should all be recorded. Only with such tracking can the long-term effectiveness of degree programs be documented. Comparisons with national data can be made with information from such sources as Cooperative Institutional Research Program's freshman survey data [7] and American College Testing's College Outcomes Measures project [8].
Step 5. Use the Assessment Results to Improve the Mathematics Major
The payoff of the assessment cycle comes when documentation of student learning and how it was achieved point the way for improvements for future students. Assessment should help guide education, so this final step in the cycle is to use the results of assessment to improve the next cycle. This is answering the third assessment question — what should be changed to improve learning? However, this important step should not be viewed solely as a periodic event. Ways to improve learning may become apparent at any point in the assessment cycle, and improvements should be implemented whenever the need is identified.
The central issue at this point is to determine valid inferences about student performances based on evidence gathered by the assessment. The evidence should show not only what the students have learned but what processes contributed to the learning. The faculty should become better informed because the data should reveal student learning in a multidimensional fashion.
When determining how to use the results of the assessment, faculty should consider a series of questions about the first four steps—setting goals and objectives, identifying learning and instructional strategies, selecting assessment methods, and documenting the results. The most critical questions are those about the learning strategies:
Secondly, questions should be raised about the assessment methods:
Finally, before beginning the assessment cycle again, the assessment process itself should be reviewed:
IV. Conclusion
During an effective assessment cycle, students become more actively engaged in learning, faculty engage in serious dialogue about student learning, interaction between students and faculty increases and becomes more open, and faculty build a stronger sense of responsibility for student learning. All members of the academic community become more conscious of and involved in the way the institution works and meets its mission.
References
[1] Guidelines for Programs and Departments in Undergraduate Mathematical Sciences, Mathematical Association of America, Washington, DC, 1993.
[2] Measuring What Counts, Mathematical Sciences Education Board, National Research Council, National Academy Press, Washington, DC, 1993.
[3] Assessment Standards for School Mathematics, National Council of Teachers of Mathematics, Reston, VA, circulating draft, 1993.
[4] Principles of Good Practice for Assessing Student Learning, American Association for Higher Education, Washington, DC, 1992.
[5] "Mandated Assessment of Educational Outcomes," Academe, November-December, 1990, pp. 34-40.
[6] Steen, L.A., ed. Heeding the Call for Change, The Mathematical Association of America, Washington, DC, 1992.
[7] Astin, A.W., Green, K.C., and Korn, W.S. The American Freshman: Twenty Year Trends, Cooperative Institutional Research Program, American Council on Education, University of California, Los Angeles, 1987. (Also annual reports on the national norms of the college freshman class.)
[8] College Level Assessment and Survey Services, The
American College Testing Program, Iowa City, 1990.
|
||
|
|
|
|