brownelearning

A few posts ago I noted that there is no Web 2.0. As Sir TB-L stated, user-to-user communication was the original intent of the web, and many us have been using it for that purpose since its inception. The only substantive difference between the web circa 1994 and Web 2.0 is that corporations have realized there is money to be made in providing a forum for user-created content. I see some strong parallels between Web 2.0 and the “universities will be irrelevant,” “they must adapt or die” chorus.

Those who insist that universities reform to embrace the Internet-age (iReformers?) assume that higher education is not already so doing. Many of us – especially, but not uniquely, in education – have long abandoned lecturing; we have shifted from memorizing facts to reasoning with them; and we have long embraced technology as learning tool. So forgive our confusion when someone says we must change to be exactly as we are, especially when we see fewer advantages and a multiplicity of disadvantages in institutionalizing our best practices.

PS – I’ll also restate this retort to those who think Higher Ed will become irrelevant: Universities existed long before they were relevant and were, in many ways, superior to what they are now. How is changing to maintain their new-found relevance a good thing?

—–

Social science in general and educational research in particular are packed with ambiguous denotations. Every time someone takes a new view on anything, they feel an urge to re-use terms from older theories; and this has left us in a disorienting clutter of nomenclature that is hardly productive. I heard two professors argue through a lunch, only to conclude that they held different meanings of the adjective “esoteric.” Does “critical” describe an issue’s importance or its research method? “Punishment,” “intelligence,” and, yes, even “validity” don’t just hold different meanings in education than in real life; they hold many distinct meanings inside education.

Compounding the collective ataxia is that terms are often employed inaccurately by less-than-qualified individuals. “Automaticity” has been reduced to “reaction time,” though it used to be a much more robust concept. “Self-efficacy” is frequently confused with “efficacy,” despite Pajares’ and Bandura’s effort to maintain a modicum of purity in their theories. I believe these two forces – the preponderance of pseudo-neologisms (old words with new meanings) and collective frowsy phraseology – hinder our field.

With this understanding I coin my own addition to our field’s jargon. I feel no hypocrisy in so doing because, contrary to my points above, this new word a) does not change the meaning of an exiting term, and b) serves to constrain its meaning to its theoretical origins.

Five decades of philosophy on the concept of validity-in-assessment has yielded several meaningful changes to how the term “validity” should be used. Specifically, it is not an assessment that is valid, nor can validity be assigned to the assessment’s results; it is the specific manner in which we place meaning on the assessment results (our inferences or interpretations) that can be more or less valid.

There are two massive practical problems with this view: First, it takes a lot of breath to refer to the “validity of the interpretations of assessment results,” so many au courant experts still call specific tests valid or invalid. It’s just their shorthand reference to validityoftheinterpretationsofassessmentresults. Second, because the great validity theorists saw themselves as refining the existent term, they had no need to add modifiers or create other variations of the word “validity.” (e.g. We don’t hear Messickian validity contrasted with Cronbachian validity, etc.)

VIAR, a pronounceable replacement for “validity of the interpretations of assessment results,” overcomes these two deficiencies. It is easier to type and say (“vi-yer”) than the phrase it represents, and its acronymous state encapsulates the parsimony and coherence of the modern view of validity. Additionally, it does not aggravate either condition I describe above: It is a new term (not a redefinition of an old word) and remains true to its founding tenets.

I don’t know if the assessment community will embrace VIAR in their speech or writing, but doing so (or finding another suitable term) would dispel much confusion.

I became re-interested in Multiple Intelligences (MI) theory a couple of days ago when a friend posted this link to his Facebook:

http://www.youtube.com/watch?v=tDtZEpf_SJ4

It’s Dr. Howard Gardner explaining the last 25 years of work on MI – a term and theory he invented. I admit I’ve only watched the first third, but I was summarily unimpressed with the general tautology of his descriptions. Further, after I heard his “Scientific Claims” of MI, I had to question whether he even understood the meaning of “scientific.” I wasn’t sure if I was correct, crazy, or just just too proud to accept the theory on which much of my colleagues’ work is based. So I turned to the literature.

A quick search found several empirical and theoretical papers that echoed my thoughts. Curiously, it seemed the Canadians held the most disdain for MI. Two of the articles (Klein, 1998 and Visser et al., 2006) included rebuttals from Gardner himself, followed by responses from the original authors. In both cases Gardner accused the researchers of not truly understanding MI. Visser responded beautifully (while backing up my observation of MI’s non-science status):

Gardner states that we have failed to grasp the core of MI theory, and perhaps in some sense he is right: it remains unclear to us what it is that MI theory can explain about intelligence, above and beyond what has already long been known. Gardner could clarify this “core” for us, by providing falsifiable, testable, MI-based hypotheses that would predict results different from those predicted by existing models of the structure of mental abilities. We encourage Gardner to provide “intelligence-fair” measures for his eight “intelligences” – tasks involving no extraneous personality, emotional,or sensory acuity content – so that MI theory can again be put to the test.

Because I’m struggling to get my head around it all.

1. The Wiki for Evaluation and Assessment: I got fed up with textbooks, so my students are writing their own this semester. Teams of students have been assigned two chapters to write and provided access to my personal library of assessment texts. They’re expected to post a finished draft for each chapter the week before we cover it in class. I’m writing the first three chapters. The first chapter on reliability they read (and critiqued to shreds) this week; it needs severe revision. The second and third chapters (correlation and validity) are in the works and will be done this weekend.

2. Completing a review of the video analysis literature. We’re trying to use standardized video experiences to replace a small portion of field experience. We hope to overcome the paradox of using non-standardized field experiences to fulfill standards. We’ve been blocked by those who fear the state would never allow it, but I just learned that the new state commissioner of education, David Stiener, has a background in video analysis of teachers. Also, I’ll be meeting with Michael Preston of Teachers College at TEDxNYED in March. I need to get something together because my abstract was accepted to EDMEDIA Toronto in June.

3. I’m on the College Technology Council, which has sent representatives (including myself) to form a working group to decide what to do about our CMS going away. (It was bought out by Satan, I mean, Blackboard.) I’m on a committee for that working group that is assessing how faculty have used the CMS in the past (to form criteria for the next CMS), and a sub-committee that is looking into emerging technologies. Yes, it goes Council→Working Group→Committee→Sub-Committee.

4. I am now the chair of the unit assessment committee. That’s the group that design the data collection for all of the college’s teacher education accreditation needs. We’re currently developing a more robust system to assess the graduate-level teacher education outcomes.

5. I’ve completed an analysis of foreign language enrollments based on government databases. I know it was wrong, but I did it without completing a lit review. Now I have to go back and finish that before I can submit the paper.

6. I’m conducting a lit review of language students’ motivation for another paper based on six years of national surveys submitted by Middle East language learners.

7. SUNY Brockport and CUNY Cortland won a Title Vi grant to increase globalization instruction at the GE level. They’re including a dual-site Chinese course, so I promised to hook them up with the efforts in Utah (BYU-Utah Hindi, and BYU-UVU Chinese). I also owe the PI an email explaining my concerns with their rather arbitrary comparison level on two groups of students…

8. The SSRC wants to continue our relationship, which is great, but I’m very busy. I’ll respond to their email this weekend when I get a moment to breathe.

9. A dear friend has asked me to check out his work on Brain Honey. That should be interesting.

10. Oh, and somewhere in there I’m teaching three preps this semester. We’re three weeks into the semester and one course doesn’t even have a completed syllabus yet!

11. Post next week’s EDI 419 activities for enterprising students who want to take advantage of their week off.

12. I almost forgot: I need to write an evaluation protocol for a massive grant application the NMELRC is turning in.

The New York Times reports that President Obama will call for an overhaul of No Child Left Behind to fashion it after his own “Race to the Top” initiative. (Why not then just revoke NCLB and pass laws called “Race to the Top?) Unfortunately, I fear RTT (RtT? R2T?) will just have different shortcoming, loopholes, etc.

I already see one glaring issue with RTT, which prompted me to submit this letter:

President Obama wishes to codify his “Race to the Top” program by overhauling the No Child Left Behind laws (NYT, January 31, 2010). This is a welcome attempt at fixing a broken system, but as a teacher educator trained in the issues of assessment, I have severe reservations about one aspect of the program.

Race to the Top specifies that student test scores must be used as criteria in judging teacher competence. The problem is that the state tests were not developed to measure teacher ability. While theory would suggest a link between student performance and teacher quality, reality is much more complex and nuanced.

The 1999 Standards for Educational and Psychological Testing, published by the American Educational Research Association, the American Psychological Association, and the National Council for Measurement in Education, require evidence to support each intended use of a test. Before we require states and districts to use students’ test scores to judge their teachers, we must first establish the degree to which student test scores are dependent on teacher quality and not on geographic location, socio-economic status, parental involvement, race, ethnicity, gender, and other traits that are beyond the teachers’ control.

Some readers may wonder if there is even evidence to support judging a student by their performance on the test and, in reality, many state tests do not fulfill the requirements set forth in the AERA/APA/NCME Standards. But the issues would be compounded were we to extend already unsupported interpretations of test results to include individuals who did even take the test.

I doubt it gets published, but I think I will start speaking a little louder about the evils of what James Popham has called “Second-step inferences.”