Making Technology Valuable: Stories of Meaningful Assessment

Our ADE project team worked very hard in Arizona last week to pull together some meaningful resources on Assessment. Largely due to the leadership and dedication of our leader, Jim Harmon, we have finally been published to iTunes.

http://itunes.apple.com/itunes-u/making-technology-valuable/id539362900?mt=2

Means of Assessment in Challenge Based Learning

As I indicated in Learning from Failure, despite the fact that their solutions came up short I was assured that my students actually achieved significant objectives during their Challenge Based Learning projects.  I used a number of means to measure the achievement of these objectives.

Though these varied devices, I was able to arrive at individual "grades" despite the fact that they had worked entirely within groups. It is worth noting even though there were very few "A's", I received  no complaints about grades.  The final grade was my "gestalt" evaluation based on several modes of assessment:

* Early in the process, I surprised students by sending them off with their cell phones\to record impromptu audio reflections. They were asked to describe the research they had done for guiding questions, the contributions they had or had not made to the group thus far, what others would say about their contributions, and who in the group had distinguished herself through her contributions. I strongly recommend such a mid-process exercise because it implicitly calls students to task without a scolding or pep talk.

* I put a high premium on group presentations as assessments. Groups are urged not to strain to convince us of how much they accomplished. Instead, they are instructed to describe their journeys toward their solutions, remarking on the high points and pitfalls along the way. They are told to address specific topics, such as "What would you have done differently? What would you have done with more resources or time?

*During the presentations, their peers completed a rubric, scoring them on the criteria that had been established for the presentations. I, too, completed a rubric and jotted down comments. When I was absent one day, my sub completed these forms and later I did too while watching them on video.

* The group filled out a rubric together on different aspects of their CBL. This measured their solutions, presentation, methods of testing their solutions, etc.

* They concluded with an 8-10 minute video with the following instructions:

Please put your 8-10 minute culminating video reflection in our shared dropbox file by class time on May 1 (no extensions). If you have trouble with dropbox you can use a thumb drive in a sealed envelope with your name clearly indicated.

Your individual reflection focuses on your unique contribution and reflection. You may go beyond the time limit but ten minutes of prepared thoughts should do the trick. Here were the original goals of the project. To what degree were these achieved with you? If they weren’t achieved, please clearly note whether they were the fault of the group dynamic, you personally, or the project itself.

Goals

*Students will acquire greater political efficacy.

*Students will solve a difficult challenge.

*Students will learn to develop instruments of assessment.

*Students will show initiative.

*Students will become policy entrepreneurs.

*Students will acquire greater ability to collaborate.

You should also freely reflect on your own unique experiences and contributions.

There are many aspects of this particular project that I would change the next time around, but I was very satisfied with the assessments.

The Subway Provides a Lesson about Testing

Joshua Bell

When I was at ISTE 2011, I attended Steven Kalmon's session with the dry title of "Assessing Progress toward Digital Age Learning."   Its an important topic, actually.  As employers more greatly value skills like creativity, initiative, problem solving, and collaboration with team; how do schools' assess progress in those skill areas?

Conventional testing does not remotely measure these things.  How could one use a multiple choice test to gauge a group experience in problem solving?  Kalmon set the table with a fascinating video recorded by The Washington Post.  The extraordinary violin virtuoso, Joshua Bell goes down into a D.C. subway, opens the case of his Stradivarius and begins to play exquisitely.  Folks simply pass him by.  Now of course some, might have no ear at all for the violin, but clearly most admittedly peg Bell as a street musician/beggar and simply can't "hear" the brilliance.  Check it out:

Here's the deal:  Policy makers, administrators, parents, and teachers who see all manner of "objective testing" -- the kind they have been socialized as being legitimate - are likely to be oblivious to the critical talents their students may be acquiring.  Of course the greater possibility is they are reluctant to change open up their ears because they do not design the kinds of activities to allow the talents to flourish.  Unless we continue to challenge them, they will keep plodding through the familiar routine that has gripped American education for eons.

Assessment Methods for Challenge Based Learning

Assessment through Reflection

Elaine Wrenn:

I absolutely love having kids do video reflections!  They have been such an authentic way for us to understand our students' thinking, where they are in the process, how they are feeling about the process, and what they are learning.  It's hard to describe the element that is captured in a video reflection that you don't always get in a written reflection.  It's like you can see them thinking as they speak and also hear the emotion and passion in their voices. I've always felt that reflection is an important part of the learning process for both the teacher and the students.  I don't know why it didn't occur to me to use video as one way to reflect but it is definitely in my reflection toolbox going forward.

I agree with Elaine. Audio and video reflections are an extremely effective way at getting at important aspects of learning.  What did the student learn from her mistakes?  What skills did she acquire through her collaboration.  How hard did she work?  In what respects did she show initiative and leadership?  Did she engage in activities that did not show up in the groups' presentation or final solution?

At Mercy, we are engaged in exploring ways to teach Tony Wagner's Seven Survival Skills.  I believe that Challenge Based Learning is an effective vehicle for addressing these.  Consequently, I have shared two of my recent student reflections in a recent slide presentation in order to provoke a discussion on how to effectively assess for these skills.  Here are the slides:

Assessment through Refection

After having required numerous reflections, I have found it critical to invest thought and time into developing good prompts.  Without specificity, the students sometimes drift into superficial commentary.  If the prompts are too specific, the respondents more or less treat them like a check list.  In the audio reflection below, Madison is responding to the following:

1) What concretely did you contribute to your group’s research and solution (I don’t mean suggested and idea . . . . What did you do like conduct an interview or edit a video).

2) Assess your individual contribution to the group’s in class presentation.  Describe your performance and your personal contribution to the slide show.

3) What did you personally learn from your project?

4) To what degree did you offer your best effort and maximize your talents in this project.

Madison's Audio Reflection

The consensus of the teachers who have listened to both reflections, agree that the following video piece is even more personal and authentic than the audio

I believe Elaine Wrenn's enthusiasm for video reflections is borne out by videos like this one.

Professional Development End Game

As I mentioned in The Other Side of the Coin post, I have heard repeated calls to assess CBL-- particularly our school's Dallas Team pilot.  On the one hand this is ironic, because no educational venture at our school has been more studied and transparent.  And on the other hand,  the call is premature since the majority of our teams have not even implemented their plans.  Don't get me wrong, I think CBL needs to be assessed-- but so should the rest of the curriculum.

Consequently, I am asking the teams of staff which are planning new CBL projects to participate in “CBL Pilot 2”at Mercy High.  Since we have plunged into CBL in order to address Tony Wagner's Seven Survival Skills for a New Economy, I have asked the teams to consider how those skills will be assessed by their new CBL projects.

I proposed that we help position our Curriculum Council with a framework for planning and assessing instruction for the Wagner Survival Skills.  The framework would invite assessment of CBL and other forms of instruction.  In other words, CBL would not be scrutinized in isolation.  If CBL does not accomplish what we hope to achieve then we need to change in more effective ways.  But change itself cannot be rejected merely by throwing stones at CBL.  If we are teaching the Survival Skills across our curriculum than any academic department should be able to demonstrate how they are attaining them.

You can see the "End Game" proposal to the CBL teams, below.  Next time, I will describe how we move from this to a (hopefully) serious contemplation of assessment.

End Game Proposal

Two of my students check out iPod touches.

The Other Side of the Coin

I admit to some fatigue with the PCG thing.  But I have revved up my engines for one last push after Easter Break.  Before I make that push, I want to reflect on an area of frustration.

It seems like some folks will take a swipe at CBL at every opportunity.  A portion of the skepticism and criticism is no doubt valid, but more than a tad is motivated by those who just want it to go away because it calls for changing old ways.

On the one hand, this frustrates me because it is hard to discuss the issues with CBL that I see, without opening a discussion up to the slings and arrows of those who resent anything related to our professional development efforts.  I hear myself becoming a booster rather than a problem solver.

However, my greater frustration comes from biting my tongue about seeing an elephant in the room when CBL is subjected to criticism: Why is CBL  subject to special scrutiny?  People want proof that it works and and assurance that it will not cut into more valuable activities.  I want to ask them to prove that those other activities work.  Can they show me that the students have attained mastery by all those homework assignments, lectures, and discussions?  Assessment is essential, but across the board.

I think I have fashioned a constructive way to  address this issue of assessing CBL.  That will be the subject of my next post.

---------------------------------------
Flickr CC Photo by woody1778a

Final Round of CBL Projects

My tenth grade American Government class is moving into the final phases of their Challenge Based Learning projects.  These ten teams have been challenged to "Get Teenagers to Care about Politics."

This challenge has included a couple of new twists:

1) I have really emphasized the importance of the students' gauging the effectiveness of their solutions.

2) We are planning a "Fight Apathy! Fair to exhibit our solutions to invited guests.  This event is being planned in conjunction with teams from two other classes (17 teams total).  Other staff members are also assisting with this event.

As we move toward the finish line,  I plan to blog on the following topics:

Weird, Wayward, and Wonderful Solutions
Already, I have encountered new issues which have occurred primarily as the result of the teams' enthusiasm.  Some students have gone very public with grammar errors and factual mistakes.  In addition, the challenge has produced a surprisingly rich variety of solutions which I look forward to sharing.

Presentations as Assessment
Starting tomorrow, teams will be making in-class presentations about their challenge journeys. Once again I have given the students a "Ten Commandments" to follow.  This set includes the importance of evaluating the success of their solution implementation.

Fight Apathy! Fair
It all goes down on April 13, 5:30 pm - 7:00 pm in school lobby.  Hundreds of students, teachers, friends, and relatives will be invited.  (Wish us luck!)

Group and  Individual Reflections
Following the Fair, the groups will be completing a written reflection.  Individuals will submit audio files.  I'll be using new prompts in both instances.

-------------------------------------
Flickr CC Photo by Dan Bock

CBL End Game

Leading professional development for Challenge Based Learning has been quite of experience*.  Like the CBL process itself,  I have done some zigging and zagging along the way.  Heading into the final quarter of the school year, I want to provide an opportunity for closure.  Consequently, I have prepared a "work sheet" for the staff teams who are developing challenges for the students.

CBL Work Sheet

I hope none of my colleagues take this as an assignment.  It is meant to provoke discussion within the teams so that they can trouble-shoot as many issues as possible before implementation.  I'm sure I left important considerations out of the list, but I found myself deciding which areas of Challenge Based Learning really meant most to me.

-------------------------------
*I'll be presenting on my experiences Challenging the Challengers at ISTE on June 29.

What Counts?

As I mentioned in Customizing Professional Development . . . ., I began this semester with the plan to suit the activity of each of my four professional groups to the needs of the repsective members.  I used a survey to inventory the  progress and needs of each Challenge Based Learning team.  However, a couple of issues stood out across groups:  1) How to word the challenge itself (see Actionable Challenges).  2) How much should the CBL project count?

Hmnn.......How much should the challenge count?  I posed a thought problem in response:  CBL will cultivate critical thinking skills, intitiative, problem solving, collabrative skills, and technology skills.  Those skills should count, right?    We shouldn't ignore the acquisition of these skills simply because we "teach" a course subject like history or Spanish, should we?

So I asked the teachers to reflect on what presently "counted" in their courses.  I suspect that in many cases reading comprehension, note taking, information retention, test taking skills, and written communication count for a great deal.  It's also possible that friendliness, short term memory, and producing homework (whether the student did it herself or not) also "counts."  But my intention is not to automatically cast aspersions on the way that points might count for a grade in another teacher's class.  I think it is perfectly reasonable to weigh the educational experience of CBL against the other class activities.

If CBL is causing us to talk about what "counts", CBL has already been good for our school.
-------------------------------------
"abacus" Flickr Creative Commons photo by feck_aRt_post

Baker's Dozen

We (the Mercy staff) finished the first semester with a twelfth round of professional cluster group  meetings. These PCGs are designed to train teachers in Challenge Based Learning Design and allow for the discussion of topics related to Tony Wagner's 21st "survival skills. In this one I pounded away at a topic near and dear to my heart-- assessment. 

My tech tip pointed to the word cloud application, Wordle.  Ironically, my heart was not particularly in this one because I'd never seen that much use for word clouds.  Fortunately, my peers pointed to a number of uses ranging from ice-breakers to t-shirt creation.  Another site was also noted that I will keep undisclosed so I can use it for a future tech tip, myself!

But the main focus was on this point:  As a school we are trying to cultivate Wagner's "survival skills:

✦ critical thinking/problem solving
✦ collaboration/leading by influence
✦ agility and adaptability
✦ initiative and entrepreneurialism
✦ effective oral and written communication
✦ accessing and analyzing information
✦ curiosity and imagination

How do we measure these?  Do we simply ignore the acquisition of these skills because we are only use to using other instruments like multiple choice tests for measuring knowledge gained?

My intention was to provoke reflection on these questions and tee us up for some practical engagement with this issue next semester as we make more progress with our new cross school learning plans.  I also shared some of my own experiences with assessment through presentation, narrative, rubric, and podcast reflections.  Most groups ended with a consideration of issues related to these different assessment tools.

Assessment Slides

---------------------------------------------
The photo in the screen capture of my slide is licensed through Creative Commons by tubachuck.

An Experiment with Alternative Assessments

As I reported in Experimenting with Assessments I wanted to attempt some alternate assessment options in my American Government Class this semester, so I launched an experiment during our first unit.

Alternate Assessment Plan

I hoped to place less emphasis on points earned through "objective" quizzes and tests.  So I created a plan where performance on the final test could be used to determine the unit grade if scores on the earlier quizzes demonstrated to me that studetnts were engaging with the material.  More significantly, students could opt out of the big test altogether and substitute performance on a project:

Project Alternatives

The results were pretty disappointing:

1) Only about 15% were inclined to do the projects and their numbers were halved as the deadline approached for the project.  They decided to take their chances with the test rather than invest energy in a project.

2) The two project alternatives attracted equal interest.  The slides tended to be better, but two of the students disregarded basic directions.  The surveys were consistently executed well, but only one student (who sought feed back ahead of time) actually got at the main concept ("legitimacy" that lay at the root of the assignment.

3) The grading scheme helped several students, however only in a handful of cases did the alternative grading systems make significant difference grade

I am pleased to know that a few kids really responded to the project options and got more out of the projects that simply going ahead and bombing another test.  But before I try something like this again, I need to retool.

------------------------------------------
A student's survey results pictured above.

Experimenting with Assessments

Last school year, I taught six sections of American Government. And I had very positive results with the projects I designed for the class. These resulted in multi media products, presentations, self-reflections and other unconventional means of assessment. Nevertheless, most of the assessment for the class was dominated by a variety of conventional quizzes and tests.

I ended the school year determined to shake this habit. Somewhat cautious (or perhaps practical) by nature, rather than tackle the entire course in one fell swoop, I decided to address the first unit. My first impulse, as usual, was to over-complicate everything. I tried to correct this, but I'm sure I will create plenty of confusion on the first go-round.

Here are the main features:

* Students choose from a menu of assessments.

* Within the conventional testing choices there are options.

* A project assessment (two choices) is available instead of some of the testing.

* A pass/fail pathway is available to a top grade.

* Students can stick with conventional testing if they (or their parents choose).

* Students will track their own progress toward their grades.

Possible Strengths

- The variety of assessments should allow for more individualized personal assessment.

- This should work well as a pilot-- With three classes in the Fall I will generate lots of feedback.

- I think I have come up with a couple of interesting multimedia ideas that are valid tests of authentic knowledge.

Possible Weaknesses

- By only sticking a toe in the water of alternative assessment, I may actually generate little enthusiasm for the alternatives.

- It's still likely to be pretty confusing for students, I'm afraid.

- I want to try this out early in the semester so that I can build upon it if I choose, but the earliness may only compound the confusion.

----------------------------

Please check out the plan. Feedback of any sort is welcome:

Unit One Assessment Options

----------------------------

"Studying Hard" Flickr Creative Commons photo by Dean+Barb

Re-imagining Learning in the 21st Century

Last year, Karen Cator was one of the leaders at the 2009 ADE Summer Institute (USA). She led the Challenge Based Learning piece of the Institute. She was hands-on during the process and her conversations with my project group had a terrific impact upon me.

Since then, Karen has become the Director of the Office of Educational Technology at the U.S. Department of Education. Recently Karen spoke at a MacArthur Foundation event at New York City's New School which focused on re-imagining learning for the 21st century. I have included a video of that event. Karen's remarks start at the 4:29 mark.

Karen's theme theme on this occasion was "Re-imagining Learning in the 21st Century". Three points in particular resonated with me:

*The current trends in educational technology are

- Mobility

- Learning outside of formal school

- Proliferation of digital content

- Publication going from print to online

Keys to 21st Century Assessment include:

- Measure what matters

- Embedded assessment

- Real time feedback

- Persistent learning

- Universal design

* And this great quote: "A highly effective teacher is one who is connected-- connected to the data, to the information, to the content and resources. . . connected to experts, and connected to learning environments for themselves and learning experiences for themselves. . . ."

Good stuff. Give Karen a listen. What resonates with you?

You Don't Need to Know This

During Exam Days, I strolled past a teacher and young student who were engaged in a feverish conversation. I overheard a snatch of exasperated comment from the teacher: "You don't need to know this-- It's not on the exam."

I understand the strange pressures that would cause a teacher to urge a student not to know something, and furthermore, I've conducted reviews for exams where students have expressed a real eagerness not to learn anything new or interesting which might not be on the test. But if you think about it, these kinds conversations are symptomatic of something pretty dreadful, aren't they? The assessment tool itself is limiting the range and depth of knowledge.

While it is easy to be cynical about this perversion of education, it is much more challenging to come up with more valid means of authentic assessing student understanding. I find myself at a challenging crossroads. As I reported in Raising the Bar . . . ., I am very happy with the intense evaluation process that I have developed for my students' Challenge Based Learning projects. But I still lean very heavily on conventional testing in other areas of my courses. (Ironically some fellow staffers members have the impression that I don't use books, paper, or testing at all).

But I would like to continue to ween myself from conventional testing, or at least present alternatives to it. My latest endeavor involves a collaboration with a new teacher. We will be teaching all the American Government sections next year. Ironically, we met to discuss our plans on the very same day that day I overheard the "You don't need to know this" remark.

Our intention is to guide our students in building a giant Fall election project. Right now, we are brain storming our ideas on a shared Google Doc, and she already has really stimulated my thinking with some great assessment suggestions. Ideally, I would like to allow my students to choose from a menu of activities and assessments so that the entire process is relatively self-directed.

Readers at the Drive-thru will hear more about this in the future, I'm sure. In the mean time I will welcome input at this blog, on Facebook, or through private email.

------------------------------------------
"March 6" Flickr Creative Commons Photo by lorenabuena

AP Government CBL Project -- Assessments

A very important aspect of Apple's Challenge Based Learning model is Assessment:

Challenge Based Learning follows a workflow that mirrors the 21st century workplace. Students are given enough space to be creative and self-directed and at the same time are provided with support, boundaries, and checkpoints to avoid frustration.

I emphasized the importance of this with my AP Government challenge groups. The groups set goals and logged them on their google docs. These tended to be overly ambitious and lofty, so I nudged them to make them more measurable and concrete.

At the end of the process, the groups reflected seriously on their journey. Across the board, their evaluations were insightful and interesting. Here is an except from the What Up Gov! group:

We found as we did our research that [one goal] was too broad, and our audience was unrealistic. . . . By changing our audience, we felt could make a greater impact and be more successful. We were trying to go global, but that was way too ambitious. . . . we focused on finding the most successful medium to do this. . . . We experimented with Twitter, a website, and Facebook. . . . The Twitter didn't receive many hits, and we felt it wasn't as successful as other mediums. The Facebook really took off well . . . We feel we really achieved our goal, because we got teens involved in the forums and sparked teen interest in democracy and our government. The discussions were very informed and interesting . . . .We're also a lot more educated on all the issues, because when we posted discussion topics we had to learn about the issues first. We also learned a lot from the posts of other people, allowing them to educate us, which we thought was really cool. . .

Pretty cool, indeed!

---------------------------------
"Reflection "I" Flickr Creative Commons photo by VisualAge

Searching for Authentic Learning in S-7

I am planning my challenge based social studies projects for the coming school year with several objects in mind. Paramount among them is striving for authentic learning.

Marilyn M. Lombardi (Duke University) says that researchers have distilled the authentic learning experience to ten design elements:

Real-world relevance
Students identification of tasks and subtasks
Sustained investigation
Multiple sources and perspectives
Collaboration
Reflection (metacognition)
Interdisciplinary perspective
Integrated assessment
Polished products
Multiple interpretations and outcomes

I think this is an exciting approach to curriculum, but I expect significant student resistance. They are conditioned to figuring out what the teacher wants, and some of the most motivated students simply want the teacher to tell them what to do, so they can do it and possibly exceed the teacher’s expectations. How will things go when I ask the students set their own expectations and complete their own assessments? Will they balk at collaboration? How much assistance will they need to critically assess their sources and develop probing questions for their investigations?

On the other hand, I think the “real world relevance of the challenge based projects may be highly motivating them. I’m sure that I will learn more than they will, next semester.You can naturally expect that my reflections on what happens in S-7 will show up at the Drive-thru in a couple of months.
--------------------------
"Authentic" Flickr Creative Commons photo by ara_p

Project Based Learning (ADE Institute Reflection #2)

When I posted Larry's Queries a couple of weeks ago, I got some helpful feedback. But one area went unaddressed:

Trickle Down Wikinomics
I teach. . .Advanced Placement U.S. Government. . . . I usually require some kind of project . . . .This time, I was thinking of assigning some wiki projects around different portions of the Constitution. . . . But I am not sure how to utilize this kind of resource with the younger students. Any ideas?

The solution came to me at the ADE Institute where we immersed ourselves in Challenge Based Learning. This concept "applies what is known about the emerging learning styles of high school students and leverages the powerful new technologies that provide new opportunities to learn to provide an authentic learning process that challenges students to make a difference." The challenge begins with a "big idea and "cascades" through a process of
*forming an essential question
*devising a challenge
*asking guiding questions
*exploring activities / resources
*determining and articulating the solution
*taking action by implementing the solution
*reflecting, assessing, and publishing.

Pardon the above jargon, but suffice it to say that this will be a huge change for me because I will be placing open ended projects in my students hands. Instead of predetermining the type of outcome I desire from them (a wiki), I will let them arrive at the solution. I'm excited, because I think they could come up with some great things, but it makes me anxious because I know they and I will be venturing far out of our comfort zones.

At the ADE Summer Institute we formed groups and engaged in a compressed version of cbl. We did not have time to produce the greatest results, but the process itself was intellectually stimulating and gave me an idea of how engaging this type of collaboration can be.

You can expect to hear back from me in a couple of months at launch.

---------------------------
Photo of Summer Institute cbl Group, by ADE '09, Kenneth Shelton