Seminar on Issues of Educational Evaluation CCT685 Spring 1999
Instructor: Peter Taylor, Critical & Creative Thinking Program
Email: peter.taylor@umb.edu; Phone: 617-287-7636
Office: Wheatley 2nd flr 143.09 (near Counseling & School Psychology)
Class Time: M 4-6.30; Office Hours: M 2.30-3.30, Tu 4.15-6.15
Course Website: http://www.faculty.umb.edu/peter_taylor/685-99.html
DRAFT, Version 13 Jan. 99

Course description
Techniques for and critical thinking about the evaluation of changes in educational practices and policies in schools, organizations, and informal contexts. Topics include quantitative and qualitative methods for design and analysis, participatory design of practices and policies, institutional learning, the wider reception or discounting of evaluations, and selected case studies, including those arising from semester-long student projects.

Sections to follow
Overview
Assessment
Basic course protocols/expectations
Texts and Materials
Schedule of classes
Additional sections in course reader
TBA
Other links
Overview
Evaluation stands for a vast set of topics, especially when we add the dimension of critical thinking about the how, what, when and why's of evaluation. Moreover, students taking this course will have a range of interests and background. What then can be achieved in one course?
The texts I have chosen provide a wealth of experience, insight and technical knowledge, so I plan to lead you to read and digest them mostly out of class. Class meetings can then revolve around a series of exercises and mini-projects through which you work some of that experience, insight and technical knowledge into your own practices. First, you will break open evaluations into their stages, then investigate the design of an evaluation of a topical educational intervention, and then investigate the wider array of influences shaping the evaluation (or lack thereof) for a topical policy change. This work will involve collaboration in groups or pairs and reporting back to the class. Finally, you will work on an individual project in which you design an evaluation in an area of your own interest.
My role as a professor will be to provide a portfolio of possible cases for the projects, to coach you in analyzing and investigating the cases and in collaborating and securing assistance as you do so, and to provide technical assistance and presentations when needed. In addition, the various activities I have planned for class meetings are intended not only to further your learning about evaluation, but also to provide models of teaching/learning tools that you might adapt to your own classrooms and other situations.

Assessment
There is no Pass/Fail option. Your grade will be based on the following components, which are described in more detail in the course packet and weekly handouts:
Written assignments and presentations, 2/3
A. Evaluation exercises and mini-projects (= 5 assignments).
B. Clippings packet: A compilation of items from current magazines and newspapers; use post-its to indicate issues where more critical thinking is needed (= 1 assignment).
C. Evaluation Design Project: Design an evaluation of a change/intervention in a specific classroom or workplace practice, an educational policy, an educational institution, or a social policy. A sequence of 4 assignments (initial description, intermediate draft, complete draft, and final version (2000-3000 words) is required.
Participation and contribution to the class process, 1/3.
E. Prepared participation and attendance at class meetings, especially when you give group or individual presentations.
F. Journal/workbook with responses/notes on exercises, projects, readings, class discussions, and clippings. Collected for perusal mid-semester.
G. Minimum of two in-office or phone conferences on your assignments and projects.
H. Optional* End-of-semester Portfolio demonstrating the development of your work (*The portfolio is for "additional work" grade only--see below).

For each of the two parts of the grade, "basic work" gives you an automatic B+. To have a chance--but not a guarantee--of getting a higher grade, "additional work" is taken into account. If you do not complete the basic work, the grade is pro-rated downwards. A passing grade of C requires 50% of the assignments or items in the following:

Written assignments and presentations:
Basic work = 80% of assignments (8 of 10) marked OK/RNR, which means "OK, Resubmission Not Requested." That is, you must submit assignments, revise in response to comments, and resubmit promptly until OK/RNR. For the project report to be OK/RNR, you must have revised in response to comments on the drafts.
Additional work = Final design project will be graded.

Participation in and contribution to the class process:
Basic work = 80% (=13) of the following 16 items: Attendance and prepared participation at the 13 class meetings and two required conferences, plus keeping a journal (reviewed mid-semester). If you miss a class, arrange to find out what happened and get any handouts so you can prepare adequately to participate in subsequent classes.
Additional work = Active participation and End-of-semester Portfolio.

Rationale: Not grading the different assignments and granting an automatic B+ for the basic work is intended to keep the focus on appreciating and learning from what each other is saying and thinking. I have found that, even when the first submissions of written assignments are excellent, angles are opened up for learning through dialogue around comments. I continue to request revision, not until a certain standard is reached, but as long as the interaction can still yield significant learning (see "Learning through dialogue around written work" in the course packet).

Additional options: 1) Alternative grading system: Students can, at the end of the semester, submit to be graded their full set of assignments and revisions. (Note: Last minute, overdue assignments cannot be added at this stage.) I will assign a grade based on the best version of each assignment. Similarly, grades can be requested for participation and contribution to class process. In both cases, if the grade turns out lower than under the system above, the better grade stands.
2) Half-value for unrevised assignments: Although I would rather no-one relies on this, at the end of the semester for students below the basic level, I count assignments that were submitted more or less on time, but were not resubmitted until OK/RNR, as half value. Similarly, half value is assigned when the student attends class, but was clearly unprepared.

Basic course protocols/expectations
1. Make time, at least 6-7 hours/week, to work on the course outside class, especially when you have to collaborate with others on projects. Preferably, set aside clear block(s) of time to do this.
2. Be responsible about class activities and collaboration (pre-reading, attendance, arrival on time, discussion, contact about non-attendance and late work)--don't wait for me or others to check in with you. If you miss a class, arrange to find out what happened and get the handouts given out so you can be prepared to participate actively in subsequent classes.
3. Read guidelines and rationales given in the course packet and in other handouts. The class meeting times are often too short to explain everything (see section below on Communication before, during, and after class).
4. Use the 80% requirement in the assessment system (see above) to drop some assignments and miss some classes when you need to accommodate to competing demands from work and life in general.
5. Do assignments on a wordprocessor so you can revise them readily. Resubmit assignments when requested, responding to comments from me and other students. Submit assignments and revisions on due dates, or submit a note about when you plan to do so.
6. Bring journal/workbooks to every class, to draw from or write in during in-class activities.
7. Arrange WWW access and get an email address, either through UMB or a place, e.g., your local library, where you can use a web browser and access email during the week between classes.
8. Make suggestions about changes and additions to the course activities and materials. Support me as I experiment in developing this course.

Texts and Materials
Required texts: Patton, M. Q. (1982). Practical Evaluation. Newbury Park, CA: Sage.
Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service--included in course packet.

Course packet, in installments available from PT. Get a three ring binder to hold the packet and other xeroxed handouts.
Additional information about classes, assignments, and other tasks is provided in the course packet and handouts (which will also be posted on the course website).

You need a workbook/journal to carry with you at all times and an organized system to store loose research materials (e.g., a 3 ring workbinder with dividers and pockets, an accordion file, or file folders).

Reserve text: Stark, J. S. and A. Thomas (Eds.) (1994). Assessment and Program Evaluation. Needham Heights, MA: Simon & Schuster.
Folders of additional readings will also be placed on reserve in Healey library.

Recommended texts (also on reserve):
- as a guide to writing and revising: Elbow, P. (1981). Writing with Power. New York: Oxford Univ. Press.
- as a guide on technical matters of writing scholarly papers: Turabian, K. L. (1996). A Manual For Writers of Term papers, Theses, and Disertations. Chicago: Univ. of Chicago Press.

SCHEDULE OF CLASSES

Class 1 (2/1)
Introduction to Issues in Evaluation I
Brief Overview
Initial look at Issues in Evaluation (Gallery Walk activity)
Steps in Evaluation (Evaluation Clock)
Organize Homework exercise I (using Evaluation Clock to analyze an educational study reported in the press)
Reading (before class 2): ES

Class 2 (2/8)
Introduction to Issues in Evaluation II
Group/pair reports on Homework exercise I
Exposing and Classifying Diferent Kinds of Issues in Evaluation (Cardstorming activity)
Organize Homework exercise II (Briefings on Patton)
*A* Asmt due: Homework exercise I

2/15 No class (Presidents' Day)

Class 3 (2/22)
Tools of Evaluation I (Presentation by PT; subject TBA based on Cardstorming)
Organize Mini-project I (to investigate the design of an evaluation of a currently topical educational intervention)
*A* Asmt due: Draft Briefing on Patton

Class 4 (3/1)
Tools of Evaluation II (Presentation by PT; subject TBA based on Draft Briefings on Patton)
Organize further work on Mini-project I
*A* Asmt due: Draft Briefing on Investigation I

Class 5 (3/8)
Practical Evaluation
Student presentations on Patton
*A* Asmt due: Revised Briefing on Patton (2 copies)

3/15 No class (Spring break)
*A* Asmt due by mail: Revised Briefing on Investigation I

Class 6 (3/22)
Politics of Evaluation I
(Why this intervention and not others? Why evaluate this intervention, but not others? Selective interpretation of evaluations. Who discounts this evaluation, but highlights others?)
Organize Mini-project II (to investigate the wider array of influences shaping the evaluation (or lack thereof) for a topical policy change)
*A* Asmt due: Supplements to Briefings on Patton

Class 7 (3/29)
Politics of Evaluation II
Organize further work on Mini-project II
Initiate individual student evaluation design projects (Freewriting & sharing activities)
*A* Journals collected for perusal (returned on the same day)
*A* Asmt due: Draft Briefing on Investigation II

Class 8 (4/5)
Facilitating Change in Groups I: Focused Conversations & Supportive Listening
*A* Asmt due: Initial Project Description

Class 9 (4/12)
Facilitating Change in Groups II: Strategic Participatory Planning
*A* Asmt due: Revised Briefing on Investigation II

4/19 No class (Patriots' Day)
*A* Asmt due by mail: Intermediate Draft of Design Project

Class 10 (4/26)
Work-in-progress Presentations on Student Projects I

Class 11 (5/3)
Work-in-progress Presentations on Student Projects II
*A* Asmt due: Complete Draft of Design Project

Class 12 (5/10)
Theories of evaluation and educational research
(Action Research as an alternative to Positivist and Interpretivist approaches; From action research to Heterogeneous Re/constructions of Social Situations)
Readings:
Carr, W. and S. Kemmis (1986). Becoming Critical: Education, Knowledge and Action Research. Geelong: Deakin University Press., chapters 6 & 7QQ
Taylor, P. J. (ms.) "Constructing Heterogeneous Webs in Socio-Environmental Research" QQwith postscript

Class 13 (5/17)
Taking Stock of Course and of Change: Where have we come and where do we go from here?
(Historical Scan and other activities)
*A* Asmt due: Final version of Design Project
*A* Asmt due: Portfolio, for those doing this additional work.