Evaluation of Educational Change
CCT693 Spring 2000
Version 29 January 2000, subject to revision
Instructor: Peter Taylor, Critical & Creative Thinking Program
Email: peter.taylor@umb.edu; Phone: 617-287-7636
Office: Wheatley 2nd flr 143.09 (near Counseling & School Psychology)
Class Time: M 4-6.30; Office/phone conference hours: M 2-3.30, or by
arrangement.
Course Website: http://www.faculty.umb.edu/peter_taylor/693-00.html
General email: Emails sent to PT with "for CCT693" in the subject line will be
forwarded to all students in the course.
Course description
Techniques for and critical thinking about the evaluation of changes in
educational practices and policies in schools, organizations, and informal
contexts. Topics include quantitative and qualitative methods for design and
analysis, participatory design of practices and policies, institutional
learning, the wider reception or discounting of evaluations, and selected case
studies, including those arising from semester-long student projects.
Overview
Sections to follow in syllabus
Overview
Assessment and Requirements
Texts and Materials
Schedule of classes
Additional sections in course packet
Notes on Teaching/Learning Interactions
Learning through dialogue around written work
Rationale for the Assessment system
Before, during and after class--Critical thinking about course readings and discussions
Communication before, during, and after class
Basic course protocols/expectations
Guide to Classes, Assignments and Readings
Evaluation and On-going Development of this Course
Overview
Reflective practice--the goal of the CCT Program--implies that we
evaluate the effects of any changes we make--in curricula, other educational
practices, institutional arrangements, and so on--and learn from that
evaluation. We can also use the results of systematic evaluations to try to
influence others to adopt our innovations, recognizing that people do not
always take notice of positive evaluations--or negative ones for that matter.
One way to increase the likelihood that others will adopt changes and sustain
them is to involve them in designing the innovations in the first place and in
implementing them.
In light of these considerations, there are a number of strands to this
course:
1. Throughout the semester, but particularly in the first 5 weeks, the case
studies, class activities, assignments, reading & precis of the text, and
the mini-project are designed so you can learn concepts and techniques for
systematic evaluation and for facilitating group participation.
2. In the second half of the semester you design an evaluation of an
educational change you are interested in--here education can be construed very
broadly. You might conduct a pilot version of this evaluation and revise the
design in light of the experience. You will also have to spend time planning
the change in practice, policy, etc. unless the change you are concerned about
has already been instituted. Perhaps you will decide to facilitiate
participation of those potentially affected in planning the change and/or its
evaluation. In any case, from the start of the course, you should be thinking
about what educational change you want to focus on for the design project.
3. Starting in week 4, you will as a group undertake an "umbrella" project of
working out how best to support each other to get competent and comfortable in
evaluating and facilitating educational change. This includes planning the
remaining classes (subject to some constraints).
4. At various points we also step aside from the educational changes you are
interested in and think critically about evaluation, especially, to consider
the politics of evaluation and educational change more generally--the wider
array of influences shaping the evaluation (or lack thereof) of changes, or
shaping the impact (or lack thereof) of evaluations.
Assessment and Requirements
Your grade will be based on the components to follow, which are
described in more detail in the course packet. For each of the two parts of
the grade, satisfactory completion of 80% of assignments or items gives you an
automatic B+. For those who meet the B+ requirements, a higher grade may be
assigned following the rubric below. (See course packet for rationale and some
additional details.)
Written assignments and presentations, 2/3
A. Assignments introducing you to evaluation (5).
B. Evaluation Design Project: Design an evaluation of a change of or
intervention in a specific classroom or workplace practice, educational policy,
educational institution, or social policy. A sequence of 5 assignments is
required--initial description, notes on work-in-progress, work-in-progress
presentation, complete draft report, and final 1500-2500 words report.
Participation and contribution to the Class Process, 1/3.
C. Prepared participation and attendance at class meetings (12)
D. Journal & Clippings, collected for perusal mid-semester & end
Journal = weekly responses/notes on homework tasks, readings, projects, class
discussions, and clippings
Clippings = at least 10 items from current magazines and newspapers, with your
annotations
E. Minimum of two in-office or phone conferences on your assignments and
projects.
F. Peer commentary on another student's draft report
G. End-of-semester Process Review on the development of your work
Rubric
B+
80% of assignments satisfactory, i.e., submitted and revised until OK/RNR (8 of
10)
80% of participation items fulfilled (14 of 18)
In addition to the requirements for B+, if your work shows half of the
following qualities, you will get an A-, and almost all of them, an A:
Assignments on time, often revised thoroughly and with new thinking in response
to comments
Project innovative; work undertaken with considerable initiative
Final report detailed and professionally presented
Final report reflects on what you have learned and indicates future directions
to develop
Active participation in class
Process Review submitted, with cover note & annotations that show deep
reflection on your process of development
If you do not reach the 80% level, the grade for Written assignments &
presentations is pro-rated from B+ down to C =50% of assignments OK/RNR.
Similarly the Participation & Process grade goes down to C = 50% of
participation items.
Texts and Materials
Required:
Weiss, C. H. (1998). Evaluation. Upper Saddle River, NJ: Prentice
Hall.
Calhoun, E. F. (1994). How to Use Action Research in the Self-Renewing
School. Alexandria, VA: ASCD.
Course packet and Supplementary Readings, available from PT.
A journal/workbook to carry with you at all times and an organized system to
store handouts and loose research materials (e.g., a 3 ring workbinder with
dividers and pockets, an accordion file, or file folders).
Recommended:
- as a guide to writing and revising: Elbow, P. (1981). Writing with
Power. New York: Oxford Univ. Press (also on reserve).
- as a guide on technical matters of writing scholarly papers: Turabian, K. L.
(1996). A Manual For Writers of Term papers, Theses, and Disertations.
Chicago: Univ. of Chicago Press (in library's reference section).
On reserve in Healey library:
Patton, M. Q. (1982). Practical Evaluation. Newbury Park, CA: Sage.
(See also the compilation of chapter precis from 1999.)
Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American
Council of Voluntary Agencies for Foreign Service.
Spencer, L. J. (1989). Winning Through Participation. Dubuque, Iowa:
Kendall/Hunt
Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto:
Canadian Institute of Cultural Affairs.
Stark, J. S. and A. Thomas (Eds.) (1994). Assessment and Program
Evaluation. Needham Heights, MA: Simon & Schuster.
Mini-Project reports from Spring 1999
Evaluation Design Project reports from Spring 1999
Folders of additional readings
Schedule Of Classes
Class 1 (1/31)
Introduction to Issues in Evaluation & Educational Change
Initial look at Issues in Evaluation (Gallery Walk activity)
Overview of Course and its strands
Overview of Evaluation (through two mini-case studies):
Core issue = Test before change->change->Test after change
Surrounding Issue = How to get people to change
Additional Steps in the Evaluation Process
Critical Incident Questionnaire
Post-class reading: Read the syllabus and Notes on Teaching/Learning
Interactions in the course packet. Review the structure of remainder of course
packet.
Assignment 1 (due week 2): Use evaluation clock to analyze an educational
study as reported in a clipping from set 1 (in the course packet). For more
instructions on this and future assignments see course packet. Related reading
(in course packet): Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook.
New York: American Council of Voluntary Agencies for Foreign Service, p. 22-23
(on eval. clock) and p. 12-17, & 21 (to provide context and introduce you
to the rest of the Sourcebook).
Class 2 (2/7)
Using the Evaluation Clock to analyze an Evaluation
*A* Asmt 1 due
Feedback on Critical Incident Questionnaire
Questions on Teaching/Learning Interactions, described in the Course packet
Reports in small groups and then to whole class on analysis using Evaluation
Clock (Asmt 1)
Assignment 2 (due class 3): Precis of one chapter from Evaluation (hard
copy & emailed version)
Class 3 (2/14)
Overview of Evaluation
*A* Asmt 2 due
Digest chapters of Evaluation (in pairs)
Digest PT's comments on Asmt 1 (in pairs)
Focused Conversation on analysis using Evaluation Clock
Provisional selection of topics for mini-project
Post-class reading (in course packet): Stanfield, B. (Ed.) (1997). The Art
of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs,
6-29.
2/21 No class (Presidents' Day)
PT e/mails comments about chapter precis (by 2/18).
Task by 2/24: Email to PT about topic chosen for mini-project (from sets 1 or
2), leads found & guidance needed (or by mail to 41 Cornell St., Arlington,
MA 02474)
Work on assignment 3 (due class 4): Evaluation clock for mini-project topic
Class 4 (2/28)
Supporting each other in evaluating and facilitating educational change
(generating a group project through Strategic Participatory Planning (SPP)),
part I
*A* Revised chapter precis (asmt 2) due, to be collated, copied, and
distributed
*A* Asmt 3 due
Post-class reading:
Review SPP notes from ICA Facilitators Manual (in course packet). See also
Spencer, L. J. (1989). chaps. 5 & 7 of Winning Through
Participation. Dubuque, Iowa: Kendall/Hunt (on reserve).
Read Evaluation using the Collection of Precis as a guide &
shortcut.
Assignment 4 (due 3/14): Questions & comments on Evaluation
Schedule first conference before 3/9
Class 5 (3/6)
Supporting each other in evaluating and facilitating educational change
(generating a group project through Strategic Participatory Planning), part
II
Post-class reading:
Review SPP notes and chaps. 5 & 7 of Spencer (continued)
3/13 No class (Spring break)
*A* Asmt 4 due 3/14 by email (or by mail to 41 Cornell St., Arlington,
MA 02474)
Work on assignment 5 (due class 6): Mini-project report
Read How to Use Action Research (by 3/27)
Class 6 (3/20)
Activities to be determined by SPP in classes 4 and 5 (some possibilities
listed at end of syllabus)
*A* Asmt 5 due
*A* Journal & clippings collected for perusal
Class 7 (3/27)
Activities to be determined by SPP
*A* Asmt due: Initial Project Description
Class 8 (4/3)
Activities to be determined by SPP
Schedule second conference before 4/6
Class 9 (4/10)
Work-in-progress Presentations on Student Projects I
*A* Asmt due: Work-in-progress notes on Design Project
4/17 No class (Patriots' Day)
Class 10 (4/24)
Work-in-progress Presentations on Student Projects II
Class 11 (5/1)
Activities to be determined by SPP
*A* Asmt due: Complete Draft of Design Project (2 copies)
Make comments on another student's draft, and send them to the person by 5/8.
Class 12 (5/8)
Taking stock of course & of change: Where have we come & where do we
go from here?
Historical Scan
Designing different kinds of course evaluations
*A* Journal & Clippings collected for perusal II, with Process
Review
*A* Comments due on another student's draft
*A* Asmt due 5/15: Final version of Design Project
Some possible topics for unspecified weeks (based on previous year's
class; see descriptions of topics, activities, readings in course packet)
Tools Of Evaluation
Making sense of quantitative analyses
Complete a small statistical study to address "arithmophobia"
Establishing a statistical support network
Facilitating Change In Groups
Focused Conversations
Constructivist Listening
Action Research
School Change
Participatory Action Research
Politics Of Evaluation
Case studies of Headstart, The Coleman Report, Mass Teachers Test
Theories Of Evaluation And Educational Research
Action Research as an alternative to Positivist and Interpretivist
approaches
From action research to Heterogeneous Re/constructions of Social Situations