brownelearning

Jon Mott has a good post on combining ePortfolio efforts with cloud computing and the potential benefits of decentralizing the assessment process. While I agree with some of his points, his comments demonstrate the consistent lack of understanding many educators have of validity and reliability. This post is meant to clarify the issues somewhat, and offer an admonition as he continues to develop this idea.

(Some assessment specialists may take issue with my treatment of these constructs. I apologize if I’ve oversimplified. Please link to more appropriate explanations as you see fit.)

Validity

The 1999 Standards define validity as…

… the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests (p. 4)

This definition communicates three essential qualities of validity in assessment.

1. Validity is not absolute, but a matter of degree. It is more proper to refer to the quality and quantity of evidence supporting validity than to declare (or aim for) an assessment as valid.
2. Assessments cannot be valid. Assessments cannot be invalid. Data cannot be valid or invalid. Validity is a property of the interpretations one makes of assessment results.
3. Interpretations are derived from specific uses of the assessment results. Validity cannot be appropriately discussed without regard for how the assessment results are to be used.

With that in mind, notice that this sentence from Jon’s post only reflects one of those three points.

Student learning portfolios are essential in the movement toward more valid and authentic assessment in higher education.

These stipulations are not simply semantic; they reflect decades of theory and empirical research and yield procedural advantages over the vernacular concept of validity. Discussing assessments (portfolios included) without regard for modern validity is akin to pontificating about the cloud without understanding the network.

Reliability

Jon doesn’t mention reliability specifically in his post, but his notion of a quasi-decentralized (loosely-coupled) assessment framework for student portfolios has a prima facie effect on the reliability of the assessment results. (Notice: Assessment results can be more or less reliable, not the assessment itself.)

In classical test theory, reliability is conceptualized as the degree to which a student’s assessment results (obtained or observed score) reflects that student’s true score (which is “known only to God”). We assume that an infinite number of assessment tasks, rated by an infinite number of raters, etc., would average out to the student’s true score, […insert math here…] so we operationalize reliability as the consistency of assessment results.

Hypothetically, an assessment should yield the same result for two students of equal true score. In other words, the results should be consistent across students of like ability. It would be unfortunate for one student to “pass” and another to “fail” solely because the assessment lacked reliability.

Jon cites Gary Brown’s encouragement to give students a high degree of control over their portfolios. I would argue that the degree to which ownership of the portfolio is decentralized (accorded to each student), there will be a necessary degradation in reliability of the judgments passed on the portfolio. I say this for two reasons:

1. If two students contribute two distinctly different artifacts to their respective portfolios, one may fail simply due to the degree of difficulty in his/her selected artifacts, while another may pass due to the relative facility of his/hers. Assessment techniques exist that could mitigate this risk, but Jon make no allusions to them in his post.
2. Portfolio judges will be unable to disaggregate two distinct learning outcomes: a) the quality of work in the portfolio, and b) the ability to recognize and select quality work to include in the portfolio (and to exclude work of lesser quality). Again, students of equal ability may receive vastly different results. One may argue that the ability to judge one’s own work is important(1), but then it should be assessed distinctly from the quality of one’s work.

Conclusion

I must acknowledge that Jon recommends “loosely-coupling,” not “decoupling,” the university and assessment; I am simply warning him of the assessment minefield into which he is marching. To shift assessment practice without regard of modern assessment theory is an all-too-common folly.

—–

(1) “Si tu réussis à bien te juger, c’est que tu es un véritable sage.” (“If you are successful at correctly judging yourself, you are truly wise.”) The Little Prince by Antoine de Saint-Exupéry

This morning I reviewed a paper about giving every middle school student a laptop. This evening, that seems so unimportant.

The Boston Globe has published photos from the last few days in Iran following the disputed election. Words fail, but the Globe has chalked up a win for traditional media.

Hard liners attacked the university… Notice what they targeted?

“Defying an official ban, hundreds of thousands of Iranian supporters of Mir Hossein Mousavi demonstrate in Tehran on Monday, June 15, 2009. (BEHROUZ MEHRI/AFP/Getty Images)”

There are three assignments due at the beginning of class next Tuesday.

1. The Project Proposal: Choose one of the assessment formats we’ve covered (traditional tests, performance assessments, portfolio assessments, or another) and blog a one-paragraph description of how you will use it. Tell me the types of tasks you will include in the assignment, how long it will be, etc. You may decide to revise (or leave out) some of your learning targets. Once you’ve submitted this assignment, we will negotiate its scope.

2. The reading: pp. 114-120 and pp. 296-308.

3. The Draft Blueprint: There is an example blueprint on p. 114 of the text. You do not have to follow that example, but you must create a document that describes your assessment and contains the same level of detail as the example in the book. Be creative! Because posting documents to your blog will be difficult, submit your draft blueprint to the discussion board on Angel (“Lessons->Turn in Your Assignments HERE->Draft Blueprint”).

[Trying desparately not to be too cheesey here.]

Hi, I’m Jeremy, and this is my blog. I just finished my second year as an assistant professor here at Brockport. My specialty is assessment & evaluation, so I teach this course plus a section of assessment for special education.

Something about myself: Because the Penguins and the Red Wings are in the Stanley Cup finals, I consider myself to be an ex-hockey fan.

If you in my summer section, be sure to leave a comment below. Use your real first name so I can give you credit.