Evaluation of Educational Change
CCT693 Spring 2000

"We did make a terrible lot of mistakes...
So we had a little self-criticism, and we said, what we know, the solutions we have, are for the problems that people don't have.
And we're trying to solve their problems by saying they have the problems that we have the solutions for.
That's academia, so it won't work.
So what we've got to do is to unlearn much of what we've learned, and then try to learn how to learn from the people."

Myles Horton, founder of the Highlander Center, now located in New Market, Tennessee,
in an interview with Bill Moyers, June 5, 1981

Transcript published in Horton, M. and B. Moyers (1983). "The adventures of a radical hillbilly: An interview with Myles Horton." Appalachian Journal 9(4): 248-285. Quote from p. 259.
See also Southern Exposure, 6(1) for a history of the Highlander Center.

Course Packet
Seminar on Evaluation of Educational Change
CCT693 Spring 2000
Web version in development. Apologies for formatting glitches

Instructor: Peter Taylor, Critical & Creative Thinking Program

See syllabus at http://www.faculty.umb.edu/peter_taylor/693-00.html
Table of Contents

Notes on Teaching/Learning Interactions
** please read & bring questions you have to class in week 2**
Learning through dialogue around written work (incl. "revise & resubmit")
Rationale for the Assessment system
Before, during and after class--Critical thinking about course readings and discussions
(incl. journal, clippings, process review, course evaluation)
Communication before, during, and after class
Basic course protocols and expectations

Guide to Classes, Assignments and Readings
including
Assignment 1 on Evaluation Clock
Assignments 2 and 4 Precis of Evaluation
Assignments 3 and 5 Mini-project
Strategic Participatory Planning
Evaluation Design Projects

Evaluation and On-going Development of this Course


Notes on Teaching/Learning Interactions

Learning through dialogue around written work

I try to create a dialogue with each student around written work, that is, around your writing, my responses, and your responses in turn. I am still learning how to engage students in this, given your various backgrounds and dispositions, and my own. (My efforts to make this system work better for students are reviewed at http:// www.faculty.umb.edu/peter_taylor/citreport.html.)
Central to this teaching/learning interaction are requests to "Revise and Resubmit." The idea is not that you make changes to please the teacher or to meet some standard, but that as a writer you use the eye of others to develop your own thinking and make it work better on readers. I continue to request revision when I judge that the interaction can still yield significant learning; the request does not mean your (re)submission was "bad." Even when the first submissions of written assignments are excellent, angles for learning through dialogue are always opened up.
In my comments I try to capture where the writer was taking me and make suggestions for how to clarify and extend the impact on readers of what was written. After letting my comments sink in, you may conclude that I have missed the point. In this case, my misreading should stimulate you to revise so as to help readers avoid mistaking the intended point. If you do not understand the directions I saw in your work or those I suggest for the revision, a face-to-face or phone conversation is the obvious next step--written comments have definite limitations when writers and readers want to appreciate and learn from what each other is saying and thinking. Please talk to me immediately if you do not see how you are benefitting from the "Revise and resubmit" process.

I keep carbon copies of my comments, but when you submit revisions, please resubmit the previous version(s) with my marginal notes. Please revise and resubmit promptly. The yield for your learning is lower if you are no longer thinking about what you were at the time you wrote. I continue to request revision, not until a certain standard is reached, but as long as the interaction can still yield significant learning.
A minimum of two in-office or phone conferences are required. I want to reduce the chance that you avoid dialogue around comments on written work. Through such dialogue profound issues are sometimes opened up about one's relationship to audience and influencing others.
Read chapters 3 and 13 of Peter Elbow's Writing With Power (Oxford U. P., 1981) for a wealth of insight about the processes of sharing written work and revising with feedback. My website also has links to some Notes on writing and revising, including Freewriting suggestions (.../698-99p.html).
I encourage you to make use of class meetings and the list of others students' phone numbers to arrange pair peer sharing and commenting according to whatever terms you pre-arrange. This will enable you to expand the kinds of readers to whom you are responding and to avoid a common trap, namely, writing as if the reader is the professor who knows enough about your topic and thinking to fill in what isn't explicit or clearly stated.
In parallel with dialogue around comments on your written work, making notes on readings can be thought of as an active dialogue with others who are not physically present. Such dialogue helps you to think deeply about ways that the information you are reading, listening to, or writing about connects with and perhaps alters your course project and your work more generally.

Rationale for the Assessment system

The rationale for not grading the different assignments and granting an automatic B+ for 80% satisfactory completion* is to keep the focus on your developing through the semester. It allows more space for students and instructor to appreciate and learn from what each other is saying and thinking (see Learning from dialogue above). My goal is to work with everyone to achieve the 80% satisfactory completion level. Students who progress steadily towards that goal during the semester usually end up producing work that meets the criteria for a higher grade than a B+ (see rubric in syllabus).

* Satisfactory completion for written assignments means you must submit the assignment, revise in response to comments, and resubmit it promptly until marked OK/RNR = "OK, Resubmission Not Requested." For the final report to be OK/RNR, you must have revised in response to comments on the draft. I sometimes request revise and resubmit on final reports. If not enough time is left for revisions, I submit an incomplete grade or, if you specifically ask me to do this, calculate and submit a final grade without an OK/RNR for the report.


Before, during, and after class--Critical thinking about course readings & discussions

CCT courses aims to help students become reflective practitioners (or "practicing reflectors"). The most important goal of this course, therefore, is that you actively ask questions about evaluating changes in educational and social practices and policies--not just during class time, but all the time. In this spirit, the class meetings are designed assuming that you will have already done quite a bit of thinking, formulated questions, and connected the week's topics to previous week's topics and to your own interests and projects. Furthermore, after class you are expected to reflect on the class and integrate new perspectives into your notes, preparation for subsequent classes, and your developing projects. Various components of the course are intended to contribute to this reflection/critical thinking:

1. A Journal with weekly responses and notes on exercises, projects, readings, class discussions, and clippings. Through writing in your journal, you will be better able to weave the course material into your own thinking, and to bring your own thinking into class activities. In preparation for class, you might write in your journal a commentary on readings, or, after class, review the readings and the class activities. In either case explore, when appropriate, the relationship between your project/ interests and the readings/activities.
Journals will be collected for perusal twice during the semester. Bind together pages with post-its or otherwise indicate which bits you do not want me to look at. I want to get an overall impression of your developing process of critical thinking about course readings and discussions.

2. Clippings. To keep up with evaluations of changes in educational and social practices and policies, compile a packet of clippings and xeroxes of articles from newspapers, magazines, journals, and websites. These should be affixed onto 3 hole punch paper, so they can be collected in a binder, which will be kept in the entrance hall to the Department of Counselling and School Psychology so everyone can have access to. Write the full citation on each article, unless it is already included. Use post-its to indicate where more critical thinking is needed and to add your own reflections on specific points. Submit the clippings twice at the same time as the journal is reviewed.

3. End-of-semester Process Review. At the end of your journal include a) a reflection (1-3 paragraphs) on the process of development of your thinking and work this semester and b) 4-6 exhibits--not necessarily your best products--that demonstrate this process. Exhibits may be journal entries, free writing, drafts, etc. Explain your choices through annotations (post-its are a good way to do this) or in the reflection. One xample of a Process Review (a.k.a. portfolio) is included at the end of the Gudie to Classes, etc. Ask me if you want to see other examples.

4. End of semester Evaluations. I devote the whole of the last class to "taking stock":
a) to feed into your future learning (and other work), you take stock of your process(es) over the semester;
b) to feed into my future teaching (and future learning about how students learn), I take stock of how you, the students, have learned.
Standard evaluation forms are not very conducive to taking stock, so we will design another evaluation form for you to complete.

Communication before, during, and after class

The limited class meeting time means that we have to a) use the time efficiently, and b) keep lines of communication open out of class. The following practices should help:

Email or call me during the week if you see a problem in the readings (e.g., missing pages), the instructions need clarification, etc.--especially when others might share your concern.

Arrange to have time on campus when you can do library research for your projects or consult with me during office hours. For people who have arranged a back-to-back class schedule, this will probably mean visiting campus on another day as well as the day of classes.

We'll start class on time. Late-comers should quietly but firmly join us--don't take a seat at the back or off to the side.

Build relations with your classmates--a lot of learning and opportunities for clarification can happen when you talk and share work with peers. This will also allow you to find out what happened and to get the handouts given out if you miss a class, and so you'll be able to prepare and participate actively in subsequent classes. The break mid-class, for which we take turns providing light refreshments (see sign-up sheet), is a good opportunity for connecting with others.

Remember to drop off and collect written work on your own from my in/out trays before you leave class. This gives me more time to set up the class and talk with you before and after class.

If you are not ready to submit an assignment or revision on the due date, submit a note about when you plan to do so. I am flexible about extensions, but I need to know that you are keeping track of your work, not simply falling and feeling behind.

Complete the Critical Incident Questionnaires (CIQs). These enable me to identify areas that need to be addressed explicitly or revisited via email or in class the following week, or taught differently in the future.

Give yourself a chance to digest comments on your assignments, and don't try to squeeze in a discussion on them when we're in a rush or otherwise distracted. Instead, use office hours, phone calls, and email.

Later in the semester, when you're concentrating on your own projects, you might establish a daily check-in with a live or phone buddy to ensure that you're doing what is essential and not simply doing what has accumulated on your list of things to do. And to help you balance the divergent and convergent aspects of the research and writing process.

Office hours and phone conferences
Office hours for in-person (W 2.143.09) or phone conferences (617-287-7636): M 2.-3.30, or by arrangement
Priority is given to those who have signed up for an appointment.
If these times don't fit, please leave a voicemail with times I can call you.

Email: I respond daily to email. If you have a problem that other students may share or a general comment send the message to me with "for CCT693" in the subject line and it will be forwarded to everyone in the course. I can handle email attachments, but prefer to comment on printed copies of assignments.

To setup a UMass email account, get your student ID and then go to UL in Healey to sign up. If you want to use UMass as your link into the internet (a.k.a. Internet Service Provider), go to the Help Desk in Computer Services in the Science building and ask to set up a PPP account. You'll need to know the operating system of your home computer, the speed of your modem, and your UMass email account (user name) and password. Make sure you get the PPP software and persist in asking for help until you have the PPP software working. Note: a PPP link is no longer needed to use the library's databases. Instead you can use the proxy server at the bottom of the library's web page (www.lib.umb.edu).

If you don't have a home computer or don't have a modem to hook up to the internet, you can still access your email via the WWW at your local library, cybercafes, etc. One way is to get a free web based email account (e.g., at www.hotmail.com) and the other is to access your UMass account using a web based interface (e.g., webinbox.com). Mail can be automatically forwarded from your UMass account to another account. Ask the Help Desk in Computer Services to help you set up forwarding.

Email protocols:
Confirm receipt of emails so I know they've got through.
Download and read emails carefully--don't respond quickly while you're logged in and paying for each minute.
Don't send a message with emotional impact until you've slept on it.
Don't send a message when it's a way to avoid talking or if it would be better to talk.
"For CCT693" emails:
-be nice to each other--no flaming, no sharking, lots of praise and constructive suggestions;
-stick to business = the course, educational and social change, and its evaluation; and
-never quote anything from email outside the class without permission from the submitter.

Basic course protocols/expectations

1. Make time, at least 6-7 hours/week, to work on the course outside class, especially when you have to collaborate with others on projects. Preferably, set aside clear block(s) of time to do this.

2. Be responsible about class activities and collaboration (pre-reading, attendance, arrival on time, discussion, contact about non-attendance and late work)--don't wait for me or others to check in with you. If you miss a class, arrange to find out what happened and get the handouts given out so you can be prepared to participate actively in subsequent classes.

3. Read guidelines and rationales given in this course packet and in other handouts. The class meeting times are often too short to explain everything (see section above on "Communication before, during, and after class").

4. Use the 80% requirement in the assessment system (see above) to drop some assignments and miss some classes when you need to accommodate to competing demands from work and life in general.

5. Do assignments on a wordprocessor so you can revise them readily. Resubmit assignments when requested, responding to comments from me and other students. Submit assignments and revisions on due dates, or submit a note about when you plan to do so.

6. Bring journal/workbooks to every class, to draw from or write in during in-class activities.

7. Arrange WWW access and get an email address, either through UMB or a place, e.g., your local library, where you can use a web browser and access email during the week between classes.

8. Make suggestions about changes and additions to the course activities and materials. Support me as I experiment in developing this course (see section Evaluation and On-Going Devel. of this Course).

9. Arranging an Incomplete and making it up: If you anticipate not finishing the assignments you wish to finish (allowing for my comments and your response to them), you must prepare a schedule of when you plan to complete them. I will discuss this with you and arrive at a contract, signed by both of us at the last class meeting. Although I want everyone to complete the course, my experience is that indefinite incompletes are a drag for all concerned. Little learning takes place if and when the unfinished work is completed.
If either component on your grade is below C at the end of the semester, the incomplete contract will require you to attend the course the next time it is offered--there is no substitute for the development of teaching/learning interactions that happens in a class over the course of the semester.

Guide to Classes, Assignments and Readings

Questions for Gallery Walk -- CCT693, Class 1

1. What changes (big or small) would you like to see pursued in teaching, schools, and educational policy?





2. What changes (big or small) have you personally been involved in trying to make?





3. In what ways did anyone take stock of the effects of the changes in 2?





4. What's helped you make the changes successfully, and what's hindered you?

Omitted from website: pointers about gallery walk activity
Omitted from website: clipping on effect of smoking ban in restaurants
Omitted from website: brochure of "important information" for phonografix approach to teaching reading


EVALUATION CLOCK
Case (educational practice/policy/intervention evaluated) =
points from clipping added Qs to consider
12. Why is evaluation needed?
________________________________________________________________________<> 1. Who wants/sponsors eval? For what decisions? [What could they DO with the results?]

________________________________________________________________________<> 2. Key issues of evaluation?
________________________________________________________________________<> 3. Specific Qs to answer? [in the survey/questionnaire/pre- vs. post-test]
Info to be sought & where?
________________________________________________________________________<> 4. Methods to gather info
for the evaluation?
________________________________________________________________________<> 5. Who participates?
In what situations is info gathered?
________________________________________________________________________<> 6. How is info analyzed to
produce evidence? [e.g., which statistical tests]
________________________________________________________________________<> 7. Who needs analysis? In
what form? Who interprets
evidence? [from evidence -> meaning]
________________________________________________________________________<> 8. What happened (re: the
thing evaluated)?
How to report that?
________________________________________________________________________<> 9. Why did that happen?
________________________________________________________________________<> 10. Lessons learned by
sponsors of evaluation?
________________________________________________________________________<> 11. What do they do differently?


Critical Incident Questionnaire*

Please take about 5 minutes to respond anonymously to each of the questions below about tonight's class. Keep one copy for yourself and put the other by the door as you leave. I'll digest the responses, report back to you next week about them, and try to make changes to respond to your responses.

1. What incident/comment/reaction/quote stands out from tonight's class?




2. At what moment did you feel most:
a. engaged with what was happening?



b. distanced from what was happening?



3. What action that anyone (teacher or student) took did you find:
a. most helpful or affirming?



b. most puzzling or confusing?"



4 (Optional). Other comments?





*Adapted from Brookfield, S. D. (1995). Becoming a Critically Reflective Teacher. San Francisco: Jossey-Bass Publishers, 115 (on reserve).

Critical Incident Questionnaire, version 2*

Please take about 5 minutes to respond anonymously to each of the questions below about tonight's class. Keep one copy for yourself and put the other by the door as you leave. I'll digest the responses, report back to you next week about them, and try to make changes to respond to your responses.

1. What concrete incidents/comments/reactions caught your attention?




2a. What excited you?


b. What frustrated you?


3a. What trends do you see emerging?



b. What are the implications of this for yourself?



4a. What are your next steps?



b. What support do you want in taking those steps?




* For use later in the semester when trends are emerging. Adapted from previous CIQ in light of "Sense-making" theory (see p. 37 of course packet)

Assignment 1 Evaluation Clock

In preparation, first read and make notes on the Evaluation Clock from the Evaluation Sourcebook (see next section of Course packet). Then, choose from the set of clippings that follow one that interests you. (Some come from areas other than education.) Fill in as much of the Evaluation Clock as you can using information from the clipping. Use loose Evaluation clock sheet for the assignment and continue on the back if you run out of space on any item.

Notes:
1. For the Evaluation Clock, the case is the evaluation, not the educational or social change at stake. If the clipping doesn't involve an evaluation, but suggests a need for one, then invent what you think should be the answers to the 12 questions of the clock. When answers to steps in the Evaluation Clock are not specified in the clipping, invent them (either design them or imagine an outcome).

2. In the right hand column add questions you have about the evaluation both:
as a critical thinker (Why didn't they do X instead?); and
as a learner (I would like to know more about that fact, technique, organization, etc.).
You might also want to note, but in a separate place, the questions you have about the educational issue itself.

3. This is a clock, a circle, not a strict chain. If your answers to one question require/ allow you to revise/ refine an earlier one, that's to be expected.

4. What is the specific evaluation (or evaluations) the clipping is based on?
If more than one, choose one to focus on. Note that different kinds of evaluation are routinely conflated by teachers, administrators, and policy makers, namely, evaluations of:
students' knowledge;
the new curricular frameworks as a means to improve students' knowledge;
the performance of teachers;
the performance of schools; and
the performance of school districts.

For this assignment, separate the diferent kinds of evaluation in the clipping, choose one or more of them to analyze, and complete a separate clock for each one you're analyzing.

Omitted from web version: Excerpts from Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service, p. 22-23 (on eval. clock) and p. 12-17, & 21 (to provide context and introduce you to the rest of the Sourcebook).
Omitted from web version: Example of completed Evaluation Clock
Omitted from web version: Clippings set 1
Assignment 2 Precis of chapter from Evaluation

Think of your precis as summaries that give other students a quick start if they were to address the topic of your chapter. This assignment allows the course to cover more topics than is possible in classtime and allows you to practice helping others address the explosion of information.

The precis will be more useful if readers do not have to go and read the chapter, so, for example, rather than saying "there are 6 ways of flushing out goals," list the 6 ways.

The first version of precis should be submitted by email as well as on paper so I can send comments by email. You should revise in response before we distribute copies to everyone in the class. See an example of a precis from last year (below) or the full compilation on reserve.

"Briefings" on various topics related to research and engagement, prepared in the same spirit of giving other students a quick start on the issues, can be viewed at http://www.faculty.umb.edu/peter_taylor/briefings-TOC.html.

Assignment 4 Questions & comments after reading Evaluation
Once you have digested one chapter thoroughly, and discussed another with a fellow student, you should be in a position to read the rest of the book "actively." That is, to decide what chapters to read word for word, and what to read for overall ideas (e.g., via intro sections, headings, and conclusions). More importantly, you should approach your reading and note-taking as an active dialogue with someone (the author) who is not physically present, thinking deeply about ways that the information you are reading connects with and perhaps alters your intended course project and your work more generally.

In this spirit, assignment 4 is to submit 5 statements or questions from your reading of Evaluation (excluding the chapter for which you made a precis). These should be expressed so that other students can appreciate the context of your statements and questions--Imagine that you were going to kick off a class discussion with each statement or questions.


An Overview of Practical Evaluation --
a briefing on chapter 1 of Patton, M. Q. (1982). Practical Evaluation. Newbury Park, CA: Sage

Sections
I. Evaluation definition
II. Mandate: Evaluation must be practical
IIA. Alternatives to practicality
IIB. Components of practicality

I. Evaluation definition
1. systematic collection of info
2. about a topic incl. activities, characteristics, outcomes or personnel, programs, policies, products
3. for use by specific people
4. for some purpose incl. reduce uncertainties, improve effectiveness, make decisions, etc.

II. Mandate: Evaluation must be practical
IIA. Alternatives to practicality
prior to about 1975 methodological rigor and accuracy emphasized
but the resulting evaluations tend to be impractical in that the process:
* takes too much time, $$, administration
* requires artificial controls (limiting program change, selection of subjects, etc.)
* uses questions that are difficult to answer
* is politically unacceptable
IIB. Components of practicality
1. Utility (of product) concern for how specific decision makers will use results
2. Feasibility (of process)
* politically anticipation of various interest groups, obtaining their co-operation & forestalling potential obstructions or misuse of results
* practically minimum disruption to participants' lives
* $$ sufficient value to justify resources expended
3. Propriety (of process) fair & ethical
and then
4. Accuracy (of product)
Combining all four requires
1. situational responsiveness
(vs. alternative: standard procedures and use of deeply embedded implicit rules/ heuristics)
This requires awareness of these procedures/rules and
willingness to risk trying something different
& depends on grounding & refreshing in fundamentals =
writing proposals, working with evaluation task force, clarifying goals, design alternatives, questionnaire construction, interviewing, data analysis, making recommendations, etc. (p. 24)
2. methodological flexibility

3. multiple roles for evaluators e.g. training sponsors in evaluation and use of info

4. political sophistication incl. self-consciousness about style

5. creativity to operate under multiple constraints

Peter Taylor (peter.taylor@umb.edu)

Omitted from web version: Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs, 6-29.

Assignments 3 & 5 Mini-project

Goal: Investigate the the evaluation of a topical educational "intervention" (a change in curriculum, organization, policy, etc.) or propose a (better) design for evaluating the intervention.
This mini-project builds on the evaluation clock exercise, but takes it further in that you will need to find out more about the case, either through locating and talking to relevant people or through library research. Even if you don't complete everything you think is needed for the topic you investigate, you should get a feel for what is in store when you work on your own evaluation projects.

Steps (see syllabus for deadlines)
i) Read through possible cases (from sets 1 and 2) and choose one, or add one of your own.

ii) Use the evaluation clock framework (see Assignment 1) to fill in as much information from the clipping/description as you can (= assignment 3);

iii) Investigate the case, either through locating and talking to relevant people or through library research (ask PT for leads & guidance); and

iv) Prepare a report on the case (= assignment 5). These reports should be summaries that give other students a quick start if they were to address that issue or topic. When composing the report think about what you would have liked to have known when you started investigating the case. See sample from last year in this course packet and the collection on reserve for possible models.

Note: The mini-project issues have many dimensions and connections to other issues. The newspaper clippings are intended as an entry point into each case, so you know its significance in the world and be interested in looking at the evaluation that set of the debate. The focus of the mini-project, however, should be on thinking about the evaluation(s) of educational change that are involved in the project (or that need to be involved). If your issue concerns student assessment or testing, that is not the evaluation of educational change that is involved; instead it's the evaluation of the test.

One caveat: In the Evaluation Clock handout I ask you to add questions you have both as a critical thinker (Why didn't they do X instead?) and as a learner (I would like to know more about that fact, technique, organization, etc.). You will have more of these questions a) the more carefully you examine the specific evaluation and b) the more you learn about the larger range of positions and proponents around the case. So do not ignore b), but don't let it distract you from a).

Case studies, Set 2
(Clippings and articles mentioned below are available from PT to copy and return)

Possible cases and initial leads
Note: To allow you to choose a case that suits your interests, these cases range across different levels--from specific classroom or workplace setting, to educational policy testing, educational institutional change, and other kinds of social policy/politics.

1 Barry Kaufman's and his wife's work with their autistic son and other hard to reach children. Start with clipping from Utne reader. Borrow (or order via ILL), Son-Rise and Son-Rise: The Miracle Continues. View website www.option.org. Call 413-229-2100, fax 229-8931, or email raun@option.org to request references that systematically evaluate their efforts. This may be a sensitive issue for them, because standard evaluations may not be sympathetic to their method. Locate other sources on their work, perhaps critical. You may have to design what they think would be a sympathetic but fair evaluation of the Kaufmans' interventions.

2 Third International Mathematics and Science Study (TIMSS) relased in spring 1998 by US Dept. of Education National Center for Educ. Statistics. To appreciate the importance of this, use Lexis-Nexis to get a representative sample of responses in the media. Locate a get a copy of a 1999 article in Science that defends the study against the criticisms in Rotberg (1998) ( Science 280:1030-1031 -- I have a copy of this).

3 C. H. Rossell and K. Baker (1996) "The effectiveness of bilingual education," Research in the Teaching of English, 30, 376-385 has been used by opponents of bilinguial education. Cummins, J. (1998). "Rossell and Baker: Their case for the effectiveness of bilingual education." Currents in Literacy 1(2 (Fall)): 10, 11, 31 (see xerox) critique their study. Use Lexis-Nexis to get a representative sample of responses to Rossell and Baker in the media. Locate other critiques of this work.

4. Gerald Coles, Reading Lessons: The Debate Over Literacy (Hill & Wang, 1998) reviews the "reading wars," i.e., highly politicized debates over methods for teaching reading. Borrow and extract key references to research evaluating different approaches (see also clipping from Education Week 2 Dec 98).

5 Tenure is granted to tenure-track faculty granted after an extensive review of their work, supposedly not as a reward for that work, but because that work indicates that they will continue to contribute in research, writing, teaching, and other ways of serving the institution and society. How good an indicator/predictor/guarantee is the tenure review process of continued contributions?

6 The CCT Program joined the Grad. College of Ed. a few years ago and became part of the Department of School Organization Curriculum and Instruction (SOCI). SOCI has standard end-of-semester course evaluations forms that students fill in. In what ways and how well do these forms contribute to faculty improving their courses?

7 President Clinton announced in his 1999 State of the Union address that he wanted schools to end "social promotion." The implication is that it is better for failing students to repeat the grade (see clipping).

8 The Graduate School at UMass Boston consists of many graduate degree granting programs. Many of these programs are located in the Grad. College of Ed. (GCOE). Some years back the Dean of the GCOE divided GCOE into three departments--School Organization Curriculum and Instruction, Counselling and School Psychology, and the Doctoral Programs (a.k.a. Leadership in Education). GCOE Programs continue but, with one exception, are now part of one of these departments. What effects was this change in the organization of an educational institution (namely, GCOE) intended to achieve? How successful has it been in achieving those effects?

9 Critique of remedial reading programs by Diane McGuiness in Why Our Children Can't Read, and What We Can Do About It. New York: The Free Press, 1997.

10 The program criticized the least by McGuiness (see #9) is the Wilson Reading System. Nevertheless, the Phono-Graphix approach promoted by McGuiness' son and daughter-in-law appears to have several advantages over the Wilson system--success rate, speed, simplicity (see articles on both). What kind of evaluation might lead Wilson teachers to appreciate and incorporate the insights of Phono-graphix?

11 In Excellence and Equality, David Fetterman recommends "the reauthorization of a national center for the gifted to revitalize gifted and talented education and in the process serve as a beacon of educational excellence for the entire educational system" (p.xv). In Playing Favorites, Mara Sapon-Shevin argues that the construction of the category of giftedness not only secures these students unequal resources, but disrupts the sense of community and diversity that she considers essential for educational progress.

12 The Educational Testing Service (ETS) has developed an electronic rater (E-rater) of essay questions and will use this in the Graduate Management Admission Test. Kaplan has already prepared tips to help test-takers to impress the E-raters (see clipping).

13 The Massachusetts Comprehensive Assessment System (MCAS) is supposed "to improve the quality of teaching and learning... by identifying students and schools that may need help meeting rigorous education goals" (S. Gorrie, MTA Today, 30 Nov. 98). What evaluations, evidence, and theory support the idea that student tests lead to effective changes in teaching and learning?

14 Seymour Papert developed LOGO in the 1970s and in his 1980 book, Mindstorms, proposed that computer technology could lead to fundamental reforms in education. What were the outcomes of using LOGO in elemntary schools?

15 In October 1998 Harold Wenglinsky of the Educational Testing Service released his study on the relationship between educational technology and student achievement in mathematics (see Education Week, 1 Oct. 98). The results were mixed, and were cited in the media to support a range of positions (see Lexis-Nexis). What can be said about the effectiveness of computer use in mathematics classrooms?

16 Jane M. Healy's Failure to Connect: How computers affect our children's minds is the work of a "techno-pusher turned critic." She warns against technology co-ordinators making decisions, while leaving more experienced educators out of the loop. What evaluations and evidence support this recommendation?

17 Problem-based learning has been described as "one of the major success stories in education since the 1970s" (D. R. Woods, Problem-Based Learning, p. ix). What do evaluations and evidence tell us about how best to get teachers to shift to PBL? (Borrow Woods' book from PT.)

18 Howard Gardner suggests that people are capable of developing strengths in eight different approaches to learning and interacting with the world. What do evaluations and evidence tell us about how teachers can best mobilize or develop these multiple intelligences? (see info package from Robert Gates).

19 The effect of Headstart type pre-school programs on later indicators (e.g., high school graduation, employment, lessened criminal convictions, etc.) has been brought to the attention of Congress and used to promote additional federal funding. More detailed studies, however, have indicated that positive effects depend on many other factors, such as, parental involvment in schooling. What do evaluations and other evidence tell us about the design of pre-school programs for the poor? (See clippings and Woodhead, M. (1988). "When psychology informs public policy." Amer. Psych. 43(6): 443-454.)

20 Bob Gaudet of the McCormack Institute at UMass Boston analysed MCAS looking at how much "value was added" by different communities' school systems, based on socio-economic predictions of how the children should have done. (Contact the McCormack Institute to get a copy of his report.)

21. "A promising new drug for depression, "MK-869," failed to clear efficacy tests last year when it turned out that patients who recived a dummy pill had done almost as well as those using MK-869. Science 284:238-240, 1999.

22. Of the six goals President Bush set for education for the year 2000, none has been achieved. What does it mean for education reformers to set goals that can't be achieved? (NY Times 22 Dec. 1999).

23. "Graduate student unions do not interfere with the ability of faculty to advise and educate graduate students, according to a recent survey... conducted by researchers at Tufts University."

24. The influence of SAT tests in College Admissions is declining according to a news item in the Dec. 99 issue of NEA Higher Education Advocate. The Princeton Review claims that the SAT "fails to measure student preparedness for college." A new approach is taking into account such things as a student's socioeconomic status, race & ethnicity, and the quality of the student's high school (Oct. 99 issue).

25. In providing more than student aid, TRIO & GEAR-UP programs encourage economically disadvantaged students to attend college.

26. "Reform after reform is being introduced [in education] with little supporting research. An economist [Alan Krueger] argues that education should be a science, not an art." NY Times Education Life Supplement, Nov. 7, 1999. See also NY Times, April 20, 1999, p. C1 & C10.

27. Dissenting voices about the value of computers in every classroom. NY Times, 17 Mar. 1999

28. "The debate about [the health and legal effects of] heroin prescription [in European countries] should promote continuing assessment of the scientific evidence underpinning current treatment, law enforcement, and prevention policies..." Science 284: 1277-1278, 1999.

29. Diet and Blood Pressure: "The emphasis on sodium as the single dietary culprit is counterproductive to our significantly reducing cardiovascular risk for most of us and diverts attention from the issues we need to address." Science 281: 933-934, 1998.

30. "Changes in public health campaigns, including those against smoking, and medical advances have substantially decreased deaths" [from cardiovascular diseases]. NY Times, 6 August 1999.

31. What is known about the effects of welfare reform on education and children? MTA Today, June 7, 1999, p. 14.

32. The Edison Project, a for-profit school system, says students gain. Others say the project is most significant as a marketing phenomenon. NY Times, 7 April 1999.

33. "Learning alongside students of different racial and ethnic backgrounds significantly enhances the educational experiences of law students, according to a recent survey released by The Civil Rights Project at Harvard University." Jan. 00 issue of NEA Higher Education Advocate.

34. "Teach Next Year boosts teaching and learning at Dorchester High School and UMass Boston" University Reporter [U. Mass. Boston] Dec. 99.

35. What is the gap between teacher's and education faculty's views about educational theory and practice? Dona Kagan (SUNY Press, 1993) observed both groups and got them to respond to each other's accounts.

36. IMP (Interactive Mathematics Program) holds that "mathematics is far more interesting to students when they get to do some real thinking." See Alper at al. in Educational Leadership, May 1996.

37. Identify an evaluation that interests you from the following: "The Math Education pages at the Math Forum are an information center for teaching issues. Topics include adult numeracy, assessment, block scheduling, and other reform topics." http://forum.swarthmore.edu

38. Identify an evaluation that interests you from the following: "A series of discussions inspired by recent work on how students learn mathematics -- ideas that form the basis of the NCTM Standards. This series is intended to be an informative and sometimes provocative overview of the current research and thinking of some key researchers in mathematics education and educational psychology. Much of the work in this area indicates that the "traditional" classroom needs to be changed if more effective learning (i.e., learning that is more conceptual and less formulaic) is to take place." http://forum.swarthmore.edu/~sarah/Discussion.Sessions/Discussion.sessions.html<>
Omitted from web version: A sample of a Mini-Project report (more on reserve in Healey Library)

Strategic Participatory Planning

There are two goals for using the 4 stage strategic participatory planning (SPP) process in classes 4 and 5:

1. To introduce you to techniques and have you experience them so that you become interested in learning, experiencing, and using them more in facilitating participation in envisioning and planning some organizational/community/social/educational change.

2. For us, as a group, to plan the "umbrella" project of working out how best to support each other to get competent and comfortable in evaluating and facilitating educational change (subject to some constraints). This means planning the remaining classes and work between classes, addressing issues such as:
the sequence of classes;
which, if any, of the possible activities and readings to use (see descriptions on the next pages);
the rhythm of each class;
how to support each other taking initiative;
monitoring when we revert back to conventional student-professor relations;
etc.

The parameters within which this planning takes place are that:
* we meet on the dates listed in the syllabus, with two classes of Work-in-progress Presentations on Student Projects (presently scheduled for weeks 9 & 10), a final "Taking Stock" class;
* each of you completes an Evaluation Design project, with stages and deadlines as listed in the syllabus; and
* we operate with the assessment system (including the revise-and-resubmit process) given in the syllabus.

The pages and worksheets to follow (from the ICA Facilitators manual) will be used to lead you through the 4 stage SPP. As an example of an outcome of the first stage of the SPP, the following are the headings given to clusters of post-its during a "card-storming" workshop on the Practical vision for the spring 1999 course.

"For this course to work well, it will be or will involve:
Co-operative project      Heterogeneous                                                         
Diversity of views        collectivity          Reflection-in-collective                        
Reflection on ideas                             facilitates change        Mobilizing            
Change that liberates us  Deep reflection                                 resources             
Going into issues in                                                      from self,            
depth                                                                                           
Resources                 Practical                                       class, and            
Specific models           models                Learning from             practitioners         
Identifying of change                           Practitioners                                   
agents                                                                                          


A vignette of strategic participatory planning

Date: Tue, 23 Mar 1999
From: jnelson@icacan.ca (Jo Nelson) [an experienced facilitator from ICA Canada]
Subject: A Story

>Rob Edelmann wrote:
>Now that that's out of the way, what's happening in the world of Facilitation? Anyone had an interesting
>discussion lately, exciting meeting, or innovative use of technology for group discussion?

How about this one?
I had a wonderful trip to Iqaluit, on Baffin Island, five weeks ago. I was working with 22 Inuit school leaders from across the new territory of Nunavut. I arrived on the day of the first elections (the territory comes into being on April 1 this year). The next day we focused on schools - actually we threw out the word school because it has very Southern white associations. First they did small group conversations (so they could talk in Inuktitut among themselves) on successful learning -- looked at their own learning, and extracted the keys to learning in an Inuit cultural context from that. In the afternoon we did a workshop on "What would a "place of learning" look like that supported Inuit culturally appropriate learning. For the first time they began to pull together their images of Inuit cultural learning and academic learning, and think about how you might teach both kinds of knowledge in a culturally appropriate way. For example, teaching math by demonstrating a number of times, then the students copying the adult. (This was one of their keys to successful learning.) Or going out on the land to study astronomy.
The second day they did small group conversations on successful leadership -- traditional leadership on the land, in the communities, in schools -- and drew out the keys to Inuit culturally appropriate leadership. Then we did a vision workshop on what they want leadership to look like in a place of learning that supports Inuit culturally appropriate learning, then obstacles and strategies on the third day, ending with personal commitments.
I got comments back afterward like "I know now my dreams can come true" and "I have courage to be a leader again". It was an utter privilege to be a part of a society that is standing at a new beginning point to forge out what it means for them to be truly human in the 21st Century!
The task now for the project is to sustain the hope and momentum as people are spread out in the isolated communitites again. I hope to be a part of it again, maybe to train people in the methods.

Jo

from the (mobile) desk of Jo Nelson

"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it's the only thing that ever has." Margaret Mead
Try our web site <www.icacan.ca>
-------
Date: Fri, 28 Jan 2000

Postscript: The new Nunavut Department of Education asked us last fall to train their staff in facilitation methods so that they can involve communities in their decisions. This was a direct spin-off of the first program, as the Minister of Education was aware of and participated in part of the February event.
Jo

Some possible topics, activities, readings that might be woven into the Strategic Plan developed for the course

Note: There is no expectation that the Plan will incorporate these topics and activities.

TOOLS OF EVALUATION
Activity: Discussion with Brian White, UMB Biology Department, of White, B. et al. (1999) "The effect of students' note sheets on exam performance." Read the manuscript and formulate questions about design of educational "experiments,' their statistical analysis, and finding people from whom to get technical assistance re: design and analysis. Brian will briefly introduce his project and respond to your questions. The goal of reading is not to understand everything nor to say "I don't know enough to follow it," but to formulate questions about the evaluation process at whatever level is appropriate to you.

Activity: Presentation by PT on the basic logic of quantitative analysis and getting help in undertaking such analysis. In preparation, write down any technical questions you have from your reading or investigation.

Activity: Complete a small statistical study together to address "arithmophobia"

Activity: Establishing a statistical support network

FACILITATING CHANGE IN GROUPS
Focused Conversations
Activity: Respond to Video on students complaining to teacher via a Focused Conversation (and other means). In a focused conversation a neutral facilitator leading participants through four phases--objective, reflective, interpretive, decisional--with the same philosophy of participation as SPP.
Reading: Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs, 6-29 (in course packet) and 30-48 (additional handout).

Constructivist Listening
Activity: Supportive listening (in pairs). Supportive or "Constructivist listening," as described by mathematics educator, Julian Weissglass, is practiced at the start of professional development workshops or other meetings with the goal of helping participants think more clearly, make better sense of experience, and listen well to others. This practice grows out of "Re-evaluation Counseling," a form of peer counselling in which two people take turns listening appreciatively to each other "re-evaluate" current and past experiences, especially when they were young, of being hurt and not helped. When such experiences come to interfere less with one's current thinking and relationships, people more readily people take more initiative in educational and social change and in their day-to-day relationships with others.
Reading: Weissglass, J. (1990). "Constructivist listening for empowerment and change." The Educational Forum 54(4): 351-370.

ACTION RESEARCH
School Change
Activity: Attend the session of the Ed. Admin. action research class in which students report on their school change projects-in-progress (Thurs 6.45). (Action research speaks to the principle of evaluating in process, rather than all at once afterwards.) "One impression [from last year] was that real world considerations led the students to shift their focus or topic quite a lot before settling in on an educational practice to try to change and to evaluate. Their questions weren't so well defined either. We shouldn't feel bad that we (teacher & students) haven't got right to the formula for how to do an effective evaluation."

Activity: Presentations by Panel of collaborators in a school change project (former M. Ed. students), followed by questions and appreciations

Participatory Action Research
In Participatory Action Research (PAR) social scientists strive to empower the people most adversely affected by some issue and allow their inquiries to be shaped through on-going engagement with them. PAR lessens the pressure to evaluate and get people to take the evaluation into account.
Activity: Discussion of readings
Readings: Taylor, P. (1999). "'Whose trees are these?' Participatory Action Research in a Kenyan agroforestry project." ms.
Excerpts from Greenwood, D. J. (1998). Introduction To Action Research: Social Research For Social Change. Thousand Oaks, CA: Sage.

POLITICS OF EVALUATION
(Why this intervention and not others? Why evaluate this intervention, but not others? Selective interpretation of evaluations. Who discounts this evaluation, but highlights others?)

Activity: Discussion of Woodhead, M. (1988). "When psychology informs public policy." American Psychologist 43(6): 443-454. From the abstract: "The potential of early educational intervention to make along-term impact on the life chances of socially disadvantaged children has now been clearly demonstrated... The evidence has been... frequently cited in support of public policies on Head Start and early childhood programs in general... [L]ess attention has been paid to the sense in which long-term effects are embedded in a wider context of family, community, and school processes that affect children not only at the time of intervention but also throughout their childhood years..."

Activity: Discussion of Hunt, M. (1985). "The dilemma in the classroom: A cross-sectional survey measures the effects of segregated schooling," in Profiles of Social Research: The Scientific Study of Human Interactions. New York: Russell Sage, 51-97. This case, in which the educational practice being evaluated was segregated schooling, should serve as an introduction to the politics of evaluation. While reading note the different ways political pressures shape the origin, conduct and interpretation of the evaluation, and responses to its findings. (The list in parentheses above is just a beginning.) And also note the ways that educational changes in educational practices can have unintended consequences because of on-going social change and politics.

Activity: Visit by education scholar-activist who has evaluated the new Mass teacher test and commented on the State's lack of evaluation of the tests.

THEORIES OF EVALUATION AND EDUCATIONAL RESEARCH
Action Research as an alternative to Positivist and Interpretivist approaches
Activity: Discussion of reading
Reading: Carr, W. and S. Kemmis (1986). Becoming Critical: Education, Knowledge and Action Research. Geelong: Deakin University Press., chapters 6 & 7 (up to p. 200)

From action research to Heterogeneous Re/constructions of Social Situations
Activity: Make an intersecting strands diagram (that teases out multiple, diverse resources) for implementing some change in an educational setting. You could begin from your own position now and fill in your background and projected future steps. Or you could begin from a general teacher's position and add items specific to you on top of that.
The strands might be:
in the classroom with students
prep & planning
admin & funding
tech. & econ. devts.
critical thinking & social contextualization of the change

Reading: Taylor, P. J. (ms.) "Constructing Heterogeneous Webs in Socio-Environmental Research."

Omitted from web version: Excerpts from ICA Facilitators Manual

Evaluation Design projects

Design an evaluation of a change/intervention in a specific classroom or workplace practice, an educational policy, an educational institution, or a social policy. A sequence of 5 assignments (initial description, notes on work-in-progress, work-in-progress presentation, complete draft, and final version; 2000-3000 words) is required.
If the educational change is not one that already happening somewhere, you will have to spend some time researching and designing that change as well. If you only have a general idea about the change you seek, it's difficult to design an evaluation of it. A risk to watch out for is getting immersed in the issues and neglecting the evaluation. Remembering that the evaluation is the centerpiece of the project may help you see what aspects of the issue to focus on, and what to put aside for future exploration.
If you are ambitious, you can go beyond designing the evaluation to carrying it out, or carrying out a pilot version. Another risk to watch out for is jumping in and carrying it out before you've designed it carefully. Even if you don't carry out any evaluation, you should design it as if you were seriously intending to. This means you'll have to imagine the situation in which the educational change would be implemented, tease out what would be involved in evaluating it, and think especially about what needs to happen in your evaluation to get someone to use the results (ref: Patton and the Evaluation Clock).
If you are really ambitious, you might pull together a group of stakeholders and practice some of the facilitation of reflection and planning introduced in this course. At the very least, I hope I interest some of you in doing more training in the facilitation methods I myself are just learning/practicing with you.

Stages in developing your project

Towards initial description:
- Sign-up for office hours or a phone conference to discuss your project ideas.
- Compose an initial statement about your project (one or two paragraphs that may, several revisions later, find their way into the introduction of your report). The point is not to have your project defined straight away, but to begin and then to continue the process of defining and refining it. In preparation for writing this statement, I suggest making notes for yourself describing:
your area of interest;
the specific case(s) you plan to consider;
the more general statement of the problem or issue beyond the specific case;
how you became concerned about this case/area;
what you want to know about this case/area by the end of the semester;
what action you think someone (specify who) should be taking on this issue;
what help you foresee needing in order to do the research; and
who the audience for your research report might be.
- Then you can begin moving around the Evaluation Clock defining what you already know, what you need to investigate, and what can only be revealed by the evaluation you'll be designing. I think of the clock as something you move back and forwards among later and earlier steps while you are developing your ideas about what educational change you envisage resulting from the evaluation you're designing, who would be making that change, and what information you'd need to present to influence them to change. The technical steps (3-6) can be formulated more readily when the audience and focus of the evaluation are clear.
- Begin background library, WWW, and phone research to find out who's done what before/ who's doing what (through writing & action) that informs your project.
- Read chapters 1 and 2 from Elbow on reserve regarding the interplay of the creative and the critical in thinking and writing.

Work-in-progress notes: Pull together your thinking to date and present it in a form that will elicit useful comments from me. The work-in-progress notes might correspond to your thinking about half to two-thirds of the Evaluation Clock. I suggest that you include in the notes:
A revised statement of your project
Annotated list of readings done or proposed, contacts made, and interviews conducted**
Plans and timetable for their completion
A map of the larger suite of issues in which your specific one is embedded. See the example on following pages from a 1999 Ed. Admin. course on action research, an approach that is not so focused on evaluation, but shares the concern with achieving effective educational change.

**The primary goal in asking for annotations is for you to check the significance of the item against your current project definition and priorities. Annotations, therefore, should indicate the relevance of the item to your topic. Annotations also make you to a) compose sentences that may find its way into your writing, and b) have your citations already typed in (use the format/citation style you intend to use for your final report). Focus is more important than quantity. Don't pack or pad this with zillions of references you've found in your searches, but instead use the assignment as a stimulus to your clarifying whether and in what ways an article is relevant to your project.

Work-in-progress presentations: When students prepare for their presentations, especially when they design visual aids, and when they hear themselves speak their presentations, it usually leads to self-clarification of the overall argument underlying their research and the eventual written reports. This, in turn, influences prioritizing of research for the remaining time. These presentations will necessarily be on work-in-progress, so you'll have to indicate where additional investigation and clarification is needed and where you think it might lead you. Aim for 10 minutes presentation, followed by 5-10 minutes of discussion. We'll give feedback (see Sense-making feedback on next page), but you may wish to design your own form of feedback (evaluation). Earlier presenters can expect to be rougher than later, and we should give feedback accordingly.
There may not be time for extensive discussion, so, to allow for more feedback, the rest of us will use notecards to make suggestions.
I recommend using visual aids, of which the simplest to use are overhead transparencies. Tips: include only key words or prompts to what you're going to say; 15-20 words only on any sheet; text should be 1/2 inch high or more. Get some transparencies and borrow pens from me, or bring material to copy center to be xeroxed onto overhead transparencies. Return spare pens and overhead sheets in the envelope outside my office door.

Drafts: The complete draft should reflect your thinking about the entire Evaluation Clock. Some answers you will be getting from your investigations, and others would only be revealed by the evaluation you're designing.
Although this is a design project, not an answers/results project, a key thing I still look for in drafts is whether you Grab readers' attention, Orient them, and move through Steps so that they appreciate what you, the writer, has proposed.
I recommend reading Elbow chapters. 4 & 5 on Direct Writing & Quick Revising, then doing this for 90 minutes with the goal of completing an extended narrative outline or short draft (say 4-5 pages). After completing this, read Elbow section III on revising, and then prepare the draft of your report.
After the complete draft is ready I ask you to pair up and comment on another student's draft. Take Elbow's chapters 3 & 13 in mind when you decide what approaches to commenting you ask for as a writer and use as a commentator. In the past I made lots of specific suggestions for clarification and change in the margins, but in my experience, such suggestions led only a minority of students beyond touching up into re-thinking and revising their ideas and writing. On the other hand, I believe that all writers value comments that reassure them that they have been listened to and their voice, however uncertain, has been heard.

Final report: These should be 1500-2500 words (plus references). For the final project report to be counted as final, you must have revised in response to comments on previously submitted drafts. Allow time for the additional investigation and thinking that may be entailed.
Cite references consistently (an annotated bibliography is not needed). For a guide on technical matters of writing scholarly papers, I recommend Turabian, K. L. (1996). A Manual For Writers of Term papers, Theses, and Disertations. Chicago: University of Chicago Press.
I will keep the reports to show future students, so please make another copy to keep for yourself. I will send you a compilation of the final reports if you send me a check for xeroxing and postage.

Omitted from web version: Map of "Our learning as anti-racist leaders" (from Ed. Admin. action research course, 1999

Brenda Dervin, in the Department of Communication at Ohio State, has developed a "Sense-Making" approach to the development of information seeking and use. Given its diverse applications and implications (Dervin 1999) one might even call it a paradigm. Her thinking leads to recommendations about forms of response that presenters learn most from -- and listeners also. The format below both acknowledges different voices and facilitates connections.

Ref: Dervin, B. (1999). "Chaos, order, and sense-making: A proposed theory for
information design," in Robert Jacobson (ed.) Information Design. Cambridge, MA: MIT Press. Also available (as of 21 May 1996) at http://edfu.lis.uiuc.edu/allerton/95/s5/dervin.draft.html.

Presenter:
Commenter:

a) I appreciated...




b) I learned...




c) I wanted to know more about...




d) I struggled with...




e) I would have been helped by...




f) My project connects with this in the following way(s)...





g) I disagreed with...




h) I think the presenter should consider...

Omitted from webversion: Example of Process review
Evaluation and On-going Development of this Course

"I see my own teaching and advising as active and multifaceted processes, involving constant monitoring and steps to improve my practices." (From my application for the CCT position.) To involve you in this, I have included here the official course evaluation from the 1999 course, a blank copy of my own evaluation form, and a commentary that I prepared for a departmental review of my first year's work. I also include the report of the research I conducted during a Fall 1999 faculty seminar. That research focused on the revise and resubmit system. In addition to continuing to develop my ability to promote dialogue around written work, I plan this semester to work on the problems of alternating b/w teacher & facilitator. This problem is conveyed in the introduction to a workshop I will be running at the meetings of the International Association of Facilitators in Toronto in late April:

Alternating between teacher and facilitator: An Exploratory Workshop
Participation in a group process or workshop is easily stifled when participants perceive a facilitator to prefer some ideas and outcomes over others. When insights emerge from the participants themselves, they become more invested in the process and the outcomes. But what should teachers who use facilitation techniques some of the time in their classes do to be perceived as facilitating neutrally, not directing the class. Conversely, what should facilitators of any kind of group do when they see that a crucial insight is not emerging?
I have designed my IAF 2000 workshop so that the participants--including myself--can learn from each other's experiences and insights regarding the tension between facilitating and teaching (broadly construed), and the difficulties alternating between the two roles. The workshop consists of a series of activities for group interaction and intrapersonal reflection that should bring the experience and insight of teacher-facilitators to the surface. (A secondary aim of the workshop is that participants in their own work will explore further the interaction and reflection activities used during the workshop that were new to them.)
[For more details, see http://www.faculty.umb.edu/peter_taylor/IAF2000.html]

Omitted from webversion: Official evaluation from 1999

Omitted from webversion: Blank copy of 1999 evaluation
Issues in Educational Evaluation (CCT685) (now CCT693, Evaluation of Educational Change) [excerpted from http://www.faculty.umb.edu/peter_taylor/portfolio99-TOC.html]

Initial Goals
Although I had experience in social research and statistics, evaluation of educational change was a new area for me as a teacher. I designed the course so that I learn as much as possible by leading students to digest the texts for themselves and for each other, coaching the students in mini-projects, and facilitating participatory planning and other group processes. This last aspect would serve two functions: the syllabus could be adjusted according to students' background and interests, and students would be introduced to the larger endeavor of working with other people in implementing and improving educational changes. In this spirit, I chose texts that emphasized the relationship between evaluator and sponsor from the formulation of questions onwards needed if outcomes are to be taken up in changes in practice and policy.
The mini-projects were based on clippings and short articles I had collected concerning evaluations undertaken or needed.
I decided not to schedule a sequence of classes on quantitative methods but to encourage students to formulate questions based on the articles they were reading and to coach them in securing statistical advice from skilled practitioners.

Challenges and Responses
The first challenge was that many students thought the course would focus on assessment of students and some of them took several weeks to appreciate the actual topic--evaluation of educational changes. In response, I have arranged for the course name to be changed to Evaluation of Educational Change. I also had to help several students sort out what is going on in the name of testing and standards. Different kinds of evaluation are routinely conflated by teachers, administrators, and policy makers, namely, evaluation of: students' knowledge; the new curricular frameworks as a means to improve students' knowledge; the performance of teachers; the performance of schools; and the performance of school districts. Students also needed to more time than I had expected to get the hang of the "evaluation clock," which divides evaluation into 12 questions (from "Why is evaluation needed?" to "What do sponsors do differently as a result?") and emphasizes re-visiting earlier questions in light of later ones. I modeled this to students, but have identified a simpler case to use for future introductions.

Time spent early on orienting students to the endeavor of evaluation led me to eliminate the second mini-project on the politics of evaluation (what things are subject to evaluations and what not, which evaluations are paid attention to, etc.) and truncate this topic. The politics of evaluation is an important issue, but CCT students, I have learned, are less interested in general obstacles to innovation than they are in creating new educational approaches and finding situations in which to try them out. At the same time, the practitioners I invited to class emphasized the importance of collaboration and patience in getting one's plans implemented, and students were very impressed by these presentations. Indeed, it seems to me that the topic of evaluation is best presented in terms of students becoming teacher-researchers or practitioner-researchers. I began to consult with GCOE colleagues teaching evaluation and action research to clarify whether to push this theme further in CCT685, or whether to direct receptive students to action research courses taught by other faculty. I plan to continue this consultation.

Some students were disconcerted for a while by the diversity of mini-project topics and by the changes made to the syllabus as we went along, but from my experience with this first class I believe I can in the future minimize syllabus changes and anticipate potential sources of confusion. Some students commented on the number of evaluation and group process tools I introduced and suggested working with fewer in greater depth. At the same time, the final projects indicated that students were trying out many of the tools in their work even without extended training. I hope I can continue to introduce a range of tools, because in the future at least some students will have done other courses with me and so fewer of the tools will be new.

The method I used to acquaint students with the texts, namely, not lecturing but having students prepare briefings for each other on the different chapters, worked quite well. I realized, however, that it would be better to use a more recent text and one with a focus on evaluation in educational contexts. I have identified a strong candidate, Evaluation by Carol Weiss, but am still reviewing possibilities.

Students raised fewer questions about quantitative methods than I had expected, even when I assigned a quantitatively-oriented manuscript of a colleague and had him come to class to discuss his work with the students. I have identified among the clippings some cases that will allow me to convey when students need to secure statistical advice from skilled practitioners.

Some students in this course (and also in CCT698 and CCT601) wanted a rubric for A vs. A- vs. B+ grades and one student prepared an elaborate model. This motivated me to prepare my own streamlined version and implement it my Fall 1999 courses.

Future Plans
I have mentioned some of my future plans in the preceding section. I have a larger "to do" list stimulated by the formative and summative evaluations of students in the course, and their participation in revising the course as we went. My other major goals for the future are to:
--expand some of the clippings into well developed cases, especially in the areas of science education;
--consult with other GCOE faculty with a view to differentiating the evaluation and research courses we offer; and
--build on the mini-project of one student last spring to push for more productive forms of course evaluation in GCOE.

From "dialogue around written work" to "taking initiative" http://www.faculty.umb.edu/peter_taylor/citreport.html