Assess It

by Jim Lengel, Hunter College School of Education

If you don't test for it, why do you teach it?

If it's not on the exam, why should I learn it?

If we're not held accountable for it, why do we waste our time on it?

Last week's article looked at the nature of educational assessment and its effect on the learning of 21st-century skills and the use of technology. See A Grain of Salt. This week we take this concept a step further, and propose a scheme for assessing 21st-century skills, and employing digital technology fully in the process.

Though many schools today claim to develop 21st-century, technology-using skills in their students, few assess them with any regularity. Instead, they administer the tests they always have: the state mastery test, the CTBS or the Iowa Test, the PSAT and SAT, and the Regents. Upon the scores from these tests they measure their success. And as we learned last week, these assessments measure but a small subset of what students needed to know in the 20th century, and little or nothing of what they need to know today. Or tomorrow.

If we want to test tomorrow's skills, we need to start today. First, we need to define what those skills are; second we need to teach those skills to the students; and third, we need to see if they've learned them.

Define the Skills

You may spend several weeks or years coming up with your own list of 21st-century skills or technology competencies; or you may piggy-back on the work of colleagues who have already done this. For a good list of 21st-century skills, look to the Partnership for 21st-Century Skills; for a list of technology competencies, check out what the faculty of Hunter College has done. Either way, you need a compact, pithy list, one that students and teachers can understand easily.

Teach the Skills

There's no sense in measuring these new skills if you haven't first taught them. And given students a chance to learn them. You will find that you have been teaching many of these 21st-century skills all along -- it's just that you weren't testing them in any formal way. So somehow you must make sure that the new skills and new technologies are embedded into every course in every classroom. Before you start assessing, give the faculty a chance to adjust the curriculum to focus on the new skills.

Assess the Skills

Don't fall into the salt mine as you do this. Don't look for a single, 15-minute test that gives you a nice, neat score on 21st-century skills. Many vendors would like to sell this to you. Don't bite. One such company touts an online test of technology competency, complete with quartiles, statistics, and graphic district-wide reports. Let's look more closely at how it works. Here's one of the questions:

(I am not making this up. See for yourself. ) This is probably not what you had in mind when you considered 21st-century skills.

But without such a standardized test, how will we know if are students are learning the new skills? How will we know if we're doing better than last year, or better than our neighbors?

Consider a three-prong approach to assessment:

• Day to day assessment in the classroom
• A structured online portfolio for every student
• Comparable standard tests

Day to day, in the classroom

As you work to make sure that every course and every teacher in every classroom teaches of 21st-century skills and technologies, make sure also that they assess them. Include them in required assignments. Award grades based on their performance. So that a B+ in English represents not only knowledge of Shakespeare but ability to collaborate with a diverse group of peers.

This is not an easy task. Approach it from two angles:

1. Take your list of 21st-century skills, and make sure each one is taught somewhere. At least once, preferably several times. Fill out a form like this:

Let the faculty determine when and where the skills being taught and assessed already. This will fill in many of the blanks. For the ones that remain blank let the faculty figure out new ways to include them.

At the same time, you should take each course, and make a list of where the skills are currently being taught and assessed, getting right down to the specific assignment in the syllabus:

English 10
Think Creatively	Poetry composition assignment
Work Creatively with Others	Drama production task
Implement Innovations
Reason Effectively
Use Systems Thinking
Make Judgments and Decisions	Moral dilemma discussion
Solve Problems
Communicate Clearly	Oral presentation assignment
...and so forth

This will uncover the skills that need more attention; the faculty will need time to develop new assignments and assessments to ensure that all the skills are included across the curriculum. The important thing is that the 21st-century skills are assessed every day, and that they count in a student's grade.

Combining these two approaches will increase the probability that all students learn some of the new skills, and that some learn all of them. And that everyone pays more attention to them. But it's not a guarantee. A student could get a passing grade in all of his courses, and yet not be fully ready for the 21st century. We need something more to ensure that every student can demonstrate all the skills.

Structured Online Portfolio

Many of the 21st-century skills cannot be measured by paper-and-pencil assignments or multiple choice tests. The ability to work collaboratively with a diverse group for instance, or to communicate clearly through a narrated slide show. These kinds of skills are best assessed though video clips, or multimedia projects, submitted by students, and evaluated by standard rubrics. How do we implement something like this?

1. Go back to your list of skills.
2. Choose those that are best evaluated through the portfolio method.
3. Publish a rubric for each one that details what is expected for mastery.
4. Require that every student submit a digital document online -- a video clip, a multimedia report, a statistical analysis -- that proves his or her mastery of the skill.
5. Assign faculty members to evaluate the submissions anonymously. And online.
6. Make a complete portfolio from each student a graduation requirement.

Because the students' work products are clearly defined, and judged by teachers who did not have them in class, they become more independent measures of performance than classroom-based assessments. And because they include multimedia documentation, they can more accurately measure the social aspects of many of the 21st-century skills. And research has found that concrete rubrics can make these kinds of assessments just as statistically reliable and valid as the bubble tests we are all used to.

Not everything goes into the portfolio; only those items that are best assessed through documentary evidence. And the completion of a useful portfolio -- one that's aimed at a public audience -- can serve as a motivating force for students as they move through their school careers.

Comparable standard tests

As a complement to the previous two approaches to assessment, consider administering a summative, standardized test that includes many of the 21st-century skills. This can provide information that the classroom-based and portfolio systems cannot: to compare your school with others; and to compare your school's performance year to year.

The available tests in this genre do not cover all the skills; they sample just a few. And their limited testing formats do not enable a full assessment of some of the social and performance-based aspects of the skills. But as a complement to the others, these tests can be very useful. The good tests are not inexpensive, and go far beyond the simplistic multiple-choice style in the example shown above. Here are some approaches to consider:

College and Work and Readiness Assessment. This test was designed by The Council for Aid to Education, a national nonprofit organization established in 1952 to advance corporate support of education and to conduct policy research on higher education. It measures how students perform on constructed response tasks that require an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills. The CWRA is delivered entirely online in a setting proctored by your own faculty. It costs $40 per student, and included a complete set of statistical reports. Here's a sample of an item from the CWRA:

Introductory Material:
You advise Pat Williams, the president of DynaTech, a company that makes precision electronic instruments
and navigational equipment. Sally Evans, a member of DynaTech’s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235. Your document library contains the following materials: (the test manual contains the actual documents listed below)
1. Newspaper article about the accident
2. Federal Accident Report on in-flight breakups in single-engine planes
3. Internal Correspondence (Pat's e-mail to you & Sally’s e-mail to Pat)
4. Charts relating to SwiftAir’s performance characteristics
5. Excerpt from magazine article comparing SwiftAir 235 to similar planes
6. Pictures and descriptions of SwiftAir Models 80 and 235

Questions:
Do the available data tend to support or refute the claim that the type of wing on the SwiftAir 235 leads
to more in-flight breakups?
What is the basis for your conclusion?
What other factors might have contributed to the accident and should be taken into account?
What is your preliminary recommendation about whether or not DynaTech should buy the plane and what is the basis for this recommendation?

Which of our 21st-century skills is this measuring? How does the situation in the sample question compare with what happens in the world outside of school? Enough schools are using the CWRA that national comparisons are possible, and change over time can be measured.

The California Critical Thinking Skills Test is an old-fashioned multiple-choice test widely used in business to measure thinking skills, one of the elements on everyone's list of 21st-century skills. It was developed by qualified psychometricians, and has been used for many years. Here's a sample question:

Using the phone at her desk, Sylvia in Corporate Sales consistently generates a very steady $1500 per hour in gross revenue for her firm. After all of her firm's costs have been subtracted, Sylvia's sales amount to $100 in bottom line (net) profits every 15 minutes. At 10:00 a.m. one day the desk phone Sylvia uses to make her sales calls breaks. Without the phone Sylvia cannot make any sales. Assume that Sylvia's regular schedule is to begin making sales calls at 8:00 a.m. Assume she works the phone for four hours, takes a one hour lunch exactly at noon, and then returns promptly to her desk for four more hours of afternoon sales. Sylvia loves her work and the broken phone is keeping her from it. If necessary she will try to repair the phone herself. Which of the following options would be in the best interest of Sylvia's firm to remedy the broken phone problem?

A = Use Ed's Phone Repair Shop down the street. Ed can replace Sylvia's phone by
10:30 a.m. Ed will charge the firm $500.
B = Assign Sylvia to a different project until her phone can be replaced with one from
the firm's current inventory. Replacing the phone is handled by the night shift.
C = Authorize Sylvia to buy a new phone during her lunch hour for $75 knowing she
can plug it in and have it working within a few minutes after she gets back to
her desk at 1:00 p.m.
D = Ask Sylvia to try to repair her phone herself. She will probably complete the
repair by 2:00 p.m.; or maybe later.

As you can see, the format reminds you of the bubble-tests described in last week's article. But the content is quite different.The focus is on a subset of thinking skills, an area often included in 21st-century lists, but not often measured by the tests most of us use today. The CCTST is certainly not a comprehensive measure of what we are looking for, but it could serve as a statistically comparable sample.

The Programme for International Student Assessment, known as PISA, has since 2000 been gathering information on the relative performance of 15-year olds in more than 60 countries. It's run by the Organization for Economic Cooperation and Development, a respected inter-governmental agency. The skills and content they focus on, though not inclusive of all the 21st-century skills, certainly come closer than most of the tests in common use today. And their methodology seems to be closer to what we are looking for. Here's a sample question:

Mathematics Unit 44 : Decreasing CO2 levels
Many scientists fear that the increasing level of CO2 gas in our atmosphere is causing climate change.
The diagram below shows the CO2 emission levels in 1990 (the light bars) for several countries (or regions),
the emission levels in 1998 (the dark bars), and the percentage change in emission levels between 1990
and 1998 (the arrows with percentages).

Question 44.1
In the diagram you can read that in the USA, the increase in CO2 emission level from 1990 to 1998 was 11%.
Show the calculation to demonstrate how the 11% is obtained.

Question 44.2
Mandy analysed the diagram and claimed she discovered a mistake in the percentage change in emission
levels: “The percentage decrease in Germany (16%) is bigger than the percentage decrease in the whole
European Union (EU total, 4%). This is not possible, since Germany is part of the EU.”
Do you agree with Mandy when she says this is not possible? Give an explanation to support your
answer.

Question 44.3
Mandy and Niels discussed which country (or region) had the largest increase of CO2 emissions.
Each came up with a different conclusion based on the diagram.
Give two possible ‘correct’ answers to this question, and explain how you can obtain each of these
answers.

Makes you think, doesn't it? Compare the nature of this question with those from the state mastery test we looked at last week. Which is closer to what we are looking for? Notice that the content of the question relates to one of the 21st-century themes from the Parternship, and the skills necessary to complete the question draw from at least three from the same list.

Unfortunately, PISA is not available for purchase by schools. It uses a sophisticated sampling procedure, so that not every student takes every test; and the scoring of the open-ended questions requires time and training that most of us are not willing to invest in. But a school that wanted to compare itself, over time and across countries, could administer the PISA questions, samples of which are available online, do its own scoring, and gather some very useful information.

Triangulation

When you're sailing along the coast, you need the bearings to at least three landmarks to determine where you are. With a modern GPS, you still need the information from at least three satellites to figure out your position. So when you want to measure 21st-century skills, it may be best to assess from three points of view: in the classroom, day to day; through structured student portfolios; and with relevant standardized instruments. The combination of information form all three may help you know where you are are, and guide you toward your destination.

For further reading:

21st-Century Skills Assessment, from the Partnership for 21st Century Skills