Comments on: Assessment Concerns in Jon Mott’s “PLNs, Portfolios, and a Loosely-Coupled Gradebook” http://brownelearning.org/blog/?p=517 Assessment, Technology, Teacher Training Mon, 28 Jun 2010 13:18:20 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: jeremy http://brownelearning.org/blog/?p=517&cpage=1#comment-3581 jeremy Wed, 17 Jun 2009 17:58:34 +0000 http://brownelearning.org/blog/?p=517#comment-3581 Jon, I'm glad that you are thinking about these. I'm guilty of seeing assessment as the only concern, so I have to admit that there are other concerns that must balance against it. I trust your experience and judgment, but it’s only very recently that teacher ed ePortfolios finally came under validity scrutiny. As far as the difficulty/facility issue: 1. Every artifact in a portfolio represents a task completed. 2. Different tasks have different levels of difficulty/facility. 3. I assume the purpose of the portfolio is to assess student ability/accomplishments. 4. A student with less ability could choose artifacts that represent easier tasks, inflating their ability. While a more ambitious student could choose an artifact that represents a more difficult task, and have their ability underestimated. One way to mitigate this would be to dictate the types of artifacts that may be included, factor in degrees of difficulty, make scoring rubrics, rating scales, etc., publicly available, and so forth. But these represent a return of ownership to the institutions, which runs counter to Brown's comments. So the question is… How much structure do we put in place to assure a certain level of reliability, while still providing the students the ownership Brown recommends? Jon, I’m glad that you are thinking about these. I’m guilty of seeing assessment as the only concern, so I have to admit that there are other concerns that must balance against it. I trust your experience and judgment, but it’s only very recently that teacher ed ePortfolios finally came under validity scrutiny.

As far as the difficulty/facility issue:

1. Every artifact in a portfolio represents a task completed.

2. Different tasks have different levels of difficulty/facility.

3. I assume the purpose of the portfolio is to assess student ability/accomplishments.

4. A student with less ability could choose artifacts that represent easier tasks, inflating their ability. While a more ambitious student could choose an artifact that represents a more difficult task, and have their ability underestimated.

One way to mitigate this would be to dictate the types of artifacts that may be included, factor in degrees of difficulty, make scoring rubrics, rating scales, etc., publicly available, and so forth. But these represent a return of ownership to the institutions, which runs counter to Brown’s comments.

So the question is…

How much structure do we put in place to assure a certain level of reliability, while still providing the students the ownership Brown recommends?

]]> By: Jon Mott http://brownelearning.org/blog/?p=517&cpage=1#comment-3576 Jon Mott Wed, 17 Jun 2009 16:35:00 +0000 http://brownelearning.org/blog/?p=517#comment-3576 Jeremy- Thanks for the response both here and on my blog. I appreciate the dialogue. You can rest assured that I understand the concepts of validity and reliability. 15 hours of methods in grad school and a decade in an assessment-oriented career have ingrained them onto my consciousness. I am guilty, perhaps, of thinking of this problem primarily from a technological affordance perspective (i.e. how teachers and students interact and exchange information during the learning process) and not enough from the perspective of "modern assessment theory." I appreciate your invitation ("warning"!) to think about these issues more carefully. I am and will. I'm crafting a lengthier response which I'll post on my blog, but I'm wondering if you would clarify something for me. What do you mean by the "degree of difficulty" or "relative facility" of a student's "selected artifacts"? I'm not quite sure what you mean here. Are you referring to the technical difficulty of creating and or publishing the artifact? Thanks again for the response. I'll try to get my reaction written and posted in the next day or two. Jon Jeremy-

Thanks for the response both here and on my blog. I appreciate the dialogue. You can rest assured that I understand the concepts of validity and reliability. 15 hours of methods in grad school and a decade in an assessment-oriented career have ingrained them onto my consciousness.

I am guilty, perhaps, of thinking of this problem primarily from a technological affordance perspective (i.e. how teachers and students interact and exchange information during the learning process) and not enough from the perspective of “modern assessment theory.” I appreciate your invitation (“warning”!) to think about these issues more carefully. I am and will.

I’m crafting a lengthier response which I’ll post on my blog, but I’m wondering if you would clarify something for me. What do you mean by the “degree of difficulty” or “relative facility” of a student’s “selected artifacts”? I’m not quite sure what you mean here. Are you referring to the technical difficulty of creating and or publishing the artifact?

Thanks again for the response. I’ll try to get my reaction written and posted in the next day or two.

Jon

]]>