brown(e)learning

Because I’m struggling to get my head around it all.

1. The Wiki for Evaluation and Assessment: I got fed up with textbooks, so my students are writing their own this semester. Teams of students have been assigned two chapters to write and provided access to my personal library of assessment texts. They’re expected to post a finished draft for each chapter the week before we cover it in class. I’m writing the first three chapters. The first chapter on reliability they read (and critiqued to shreds) this week; it needs severe revision. The second and third chapters (correlation and validity) are in the works and will be done this weekend.

2. Completing a review of the video analysis literature. We’re trying to use standardized video experiences to replace a small portion of field experience. We hope to overcome the paradox of using non-standardized field experiences to fulfill standards. We’ve been blocked by those who fear the state would never allow it, but I just learned that the new state commissioner of education, David Stiener, has a background in video analysis of teachers. Also, I’ll be meeting with Michael Preston of Teachers College at TEDxNYED in March. I need to get something together because my abstract was accepted to EDMEDIA Toronto in June.

3. I’m on the College Technology Council, which has sent representatives (including myself) to form a working group to decide what to do about our CMS going away. (It was bought out by Satan, I mean, Blackboard.) I’m on a committee for that working group that is assessing how faculty have used the CMS in the past (to form criteria for the next CMS), and a sub-committee that is looking into emerging technologies. Yes, it goes Council→Working Group→Committee→Sub-Committee.

4. I am now the chair of the unit assessment committee. That’s the group that design the data collection for all of the college’s teacher education accreditation needs. We’re currently developing a more robust system to assess the graduate-level teacher education outcomes.

5. I’ve completed an analysis of foreign language enrollments based on government databases. I know it was wrong, but I did it without completing a lit review. Now I have to go back and finish that before I can submit the paper.

6. I’m conducting a lit review of language students’ motivation for another paper based on six years of national surveys submitted by Middle East language learners.

7. SUNY Brockport and CUNY Cortland won a Title Vi grant to increase globalization instruction at the GE level. They’re including a dual-site Chinese course, so I promised to hook them up with the efforts in Utah (BYU-Utah Hindi, and BYU-UVU Chinese). I also owe the PI an email explaining my concerns with their rather arbitrary comparison level on two groups of students…

8. The SSRC wants to continue our relationship, which is great, but I’m very busy. I’ll respond to their email this weekend when I get a moment to breathe.

9. A dear friend has asked me to check out his work on Brain Honey. That should be interesting.

10. Oh, and somewhere in there I’m teaching three preps this semester. We’re three weeks into the semester and one course doesn’t even have a completed syllabus yet!

11. Post next week’s EDI 419 activities for enterprising students who want to take advantage of their week off.

12. I almost forgot: I need to write an evaluation protocol for a massive grant application the NMELRC is turning in.

Something reminded me that some people still want to use an overly-inclusive definition of technology. So, I dug this out of an unpublished version of my dissertation prospectus:

Although some prefer an inclusive definition of technology in education (Seels & Richey, 1994; Driscol, 2001; Gibbons, 2003), the popular view of the term, which is shared by opponents of technology integration (Cordes & Miller, 2000), refers specifically to microcomputers and their peripherals. The broader definition is not as useful because it includes such disparate “technologies” as colored chalk (Wilkins, Peck, & Jordan, 1991) and analog clocks (Anderson, 1997). Therefore, this research will use the narrower point of view.

I joined a reading circle around Kendall’s book hoping to be somewhat initiated into the issues surrounding race relation theory and debate. I ended up understanding more than how those at the forefront of race-oriented organizational behavior think: I learned why they may think the way they do (or at least why this one author thinks the way she does). However, Kendall’s techniques for justifying her perspective (e.g. using only very “clean” data) could serve equally those who hold contrary views. Therefore with only specially selected anecdotes to support its thesis, this book fails to contribute anything beyond marching orders for those who already agreed with it.

Read the rest of this entry »

There is a wonderful exhibit at the Museum of Art on the BYU campus called, “Nostalgia and Technology: Embracing the New through Art and Design” by Dr. Marc Olivier, who, by the way, was one of my favorite undergraduate professors. The point of the exhibit is how we tame innovation, or change, through incorporating it into our memories of the past. Ornately carved sewing machines, cars that remind us of jet fighters, and typewriters with piano-like keyboards are just a few of the highlights of the most artistic history exhibit I’ve ever seen.

It may be its uniqueness, but this exhibit left with me the same feeling I got from the Gothic sculpture section of the Louvre: I have to go back and see that again.

If you go, be sure to take some time to enjoy the video projects which are timed to complement each other’s content. For example, some of you will recognize the hatching of the personality stealing pods from “The Invasion of the Body Snatchers” while on the next wall, silhouettes with white ear buds dance against a multicolored background in an MP3 player commercial.

There is more information on the exhibit on BYU’s MOA website: http://cfac.byu.edu/moa/Exhibits/nost_tech.php , and there is a podcast of Dr. Olivier’s appearance on NPR: http://www.publicbroadcasting.net/kuer/news.mediaplayer?STATION_NAME=kuer&MEDIA_ID=494949&MEDIA_EXTENSION=mp3&MODULE=news

For anyone who has bought a computer and doesn’t have to dough to fork over for MS Office, or uses multiple platforms and would like a full office suite that works on each of them, or if you’re a free/open software freak (but then you’d already know about this) – OpenOffice.org has released version 2 of their software.

OpenOffice is sponsored by Sun Microsystems, and I got to know it back when it was called StarOffice. It’s always been a bit clunky, but always functional, and I’ve never had an issue reading or writing documents in MS formats. Plus, OO can export straight to PDF, which is a huge benefit for many of us.

The new version has removed most of the clunkiness, but I am a bit disappointed that the new interface looks so much like every other suite out there. (I suppose they’re going for familiarity to help migrate users.) OO2 also uses the OpenDocument format as it default file format. You may have heard that the Commonwealth of Massachusetts, in an effort to curtail vendor lock-in, has declared that they will only use software that supports OpenDocuments. Therefore, it’s an obviously time for OO to move away from their own formats.

Anyway, installation was a breeze, as is saving/opening MS documents. Give it a try… It’s free.

A last note: For those of you who insist on pirating MS products: Stop. You are not doing yourselves, or the rest of us any favors. The more you do, the more groups like the BSA will gain power, and the harder it will be for alternatives, that are less evil, to become viable. Use OO. There’s even a torrent on their website.

The Program Evaluation Standards is a reference text written by a joint committee, with representatives sent from various organizations. It, being a reference, is rather formulaic in its format, but provides, nonetheless, insight into the evaluation field. In short, this book, like the field in which it is used, is a horse by committee: A camel.

The format is helpful, and follows the structure of the 30 standards. Each standard is grouped into one of four sections: Utility, Feasibility, Propriety, or Accuracy, and is clearly stated in bold print. Following the text of the standard is an overview, then guidelines. The most helpful part of each standard is the “Common Errors”, and then there follows two or more narrative descriptions of actual evaluations and an analysis of that situation.

While some might find the prose of the narratives to be helpful, I kept thinking of my American Heritage text (back in my undergrad), where there was more fluff than anything else. While these narratives are not fluffy (they are quite concise) they do simply show examples of what has already been stated in the guidelines and common errors.

The text’s insights into the evaluation domain are probably unintentional. For example, under the clarity standard (U5), the overview refers to, “narratives, illustrations and descriptions,” which explains Patton’s approach, as well as the narrative section of each standard. However, I have to wonder which came first: The people who like to use narratives or the standard to use them.

Now then, I said these standards, reflecting the field, were a camel, and I better back that up. Simply put, this book should be subtitled: If it’s light gray, do X. If it’s dark gray, do Y. The contextual nature of today’s evaluation field (even the situational appropriateness that Patton advocates) creates a world of relativism, where nothing, even standards can be set in stone. For example, under the the service orientation standard (P1), a common error is recommending termination of a program without seeing how it may be improved, but the very next error is failing to recommend termination of an ineffective program. (In fairness, I showed this to my professor, and he claimed that the first error referred to formative evaluations while the second to summative evaluations. He did comment that this should be explicitly stated in the text.)

Finally, almost every standard contains, as a final common error, the tendency to overdo the standard. For example, the first standard impresses upon the reader the importance of identifying every possible stakeholder, and gives various methods for so doing. Then it cautions to be wary of time and resource limitations. Basically, look for stakeholders, but don’t spend all your time doing it.

(I guess this book reads like the Pirate Code from Pirates of the Caribbean: “Here’s how you should always things, but they’re more like guidelines than rules.”)

As for me and evaluation, do I care about it? Yes. I care about evaluation as follow-up to all the work I do. But, as a field, evaluation seems to me to be quite immature. While there are difference of opinion in other domains of instruction science, the disputes do not, as they do in evaluation, focus around what the domain is and what its practitioners should and should not morally do. I blame the time period during which evaluation has grown up for its seemingly endless caveats and negotiations. This text supports this assertion very well.

Utilization-focused Evaluation was an interesting read for me. If you follow the phenomenology approach to literary criticism, you will know why I appended “for me” to that opening sentence. I am enormously grateful that I checked this book out of the BYU library rather than purchasing it, as it is filled with some good advice, some false doctrine, but ultimately self-aggrandizing, situationally appropriate drivel.

Imagine, if you will, a somewhat young PhD student in Instructional Psychology & Technology who wished to enter academia as a measurement expert. He sees that he needs to take two evaluation courses, so he puts them off until after his indoctrination into the cult of methods is completed. As part of his first evaluation course, he reads a book which begins with good information on proper, common sense evaluation techniques, but then slowly slides into qualitative propaganda. Such was my experience with this book.

I dedicated myself to reading the text in its entirety, not the typical, read all the headings, look at all the tables and memorize all the lists approach. Sadly, that seems to be the most effective way of taking in this book, since most of the prose is the author sharing his personal experiences that are supposed to prove his points. Oh, how I learned to detest the words, “An example might help get this point across.”

The only chapter worth reading from first page to last is the 40- page review of the qualitative/quantitative debate. The author puts on a facade of “fairness” (which he defines later in the book) by presenting the debate in a point-counterpoint format. The only issue is that all of his “points” are from the quantitative camp, and his “counter-point” salvos are qualitative. I’ve seen this technique in creationist and Jehovah Witness literature. Most likely, qualitative enthusiasts will devour this section while quantitative scientists will shake their collective heads.

He follows that chapter with a proposition for an eclectic approach. He proudly declares, “There are no logical reasons why qualitative and quantitative methods cannot be used together,” (277) without mentioning that, by the way, there are fundamentally conflicting assumptions in the approaches. Dr. Yanchar would say, “How can you use both to find “truth” if they define “truth” differently?”

Going back to the beginning of the book, he spends a few too many pages lamenting the lack of appreciation the general public has for evaluators. He claims this is because evaluations have not been properly focused on getting themselves used, but later he talks about blurring the line of what an evaluator’s responsibilities are, simply because the funder thinks it should be something else:

“In each of the examples I’ve shared [where he did something some would not consider part of evaluation], and there are many others, my participation, identity, and role were considered evaluative by those with whom I was engaged (and by whom I was paid).” (109)

Could it be that this very chameleon behavior by evaluators is what leads the public to under esteem evaluation as a whole?

Finally, after three hundred pages of giving rules for effective evaluations, and then showing appropriate violations of those rules, the author states that “there are no absolute rules.” (367) That statement pretty much sums up the entire book, and, in this reviewer’s mind, the entire evaluation field. It seems to suffer from such variance and eclecticism that even the practitioners cannot come to a clear conclusion of what they’re supposed to do.

In short, if you are required to read this text for a class, check it out of the library, as you would most likely sell it back anyway, read the headings, the first sentence of each paragraph, note the lists and the bold words, and review the tables. Don’t waste your time reading the whole thing. Trust me, I did.

Every so often, I get the real treat of reading something by someone who thinks like I do. That’s not to say that thinking like me is hard, or necessarily of value, but it is nice to come into contact with kindred spirits. The Hacker Ethic by Pekka Himanen, a book I really should have read a long time ago, not only gives me plenty to agree with, but also explains why I think the way I do. This promotes intensive nombrilism, which may lead to either a strengthening of said beliefs, or a complete re-evaluation of where I stand. In either case, The Hacker Ethic lays out, essentially, what makes geeks tick (their beliefs, priorities, and motivations), how these traits have developed, and how they fit in with today’s world.

A personal, probably apocryphal note: My father once told me that when I was a preteen, I had just said something (probably rather intelligent, but nonetheless) off the wall, and my older sister asked in frustration, “Why does he have to be such a geek?” To which my father replied something along the lines of, “Don’t complain, geeks grow up to make a lot of money.” My father’s prophesy on geeks in general proved mainly incorrect (and very incorrect in my case). One of the things made clear in this book is that the only rich geeks out there (or, the only ones who flaunt being rich), are those who long ago abandoned their geek status for more worldly pursuits.

The book begins with description of the Protestant Work Ethic, and what it means in today’s society. Among other things, this ethic values work as an end, not just a means. Unfortunately, this requires all other priorities to be squeezed out of our life, or slowly turned into slightly more pleasant versions of work. One poignant example is Himanen’s observation that we can no longer “hang out” with our kids. We require “quality time,” which must have some “goal” that can be “evaluated” at the end of the time allotted. Also, just as companies (work) have learned to contract out tasks to specialized firms and laborers, we now have day care and three-minute microwave meals.

From there the reader is bounced back and forth between analogies, similes, metaphors, and all those other literary techniques you don’t remember from sophomore English. Most of these contrast the new, intriguing Hacker Ethic to the Protestant Work Ethic of the last century, comparing, for example, their respective origins in academic and monastic life.

The explanation shifts suddenly to the ideals of the Hacker community including free speech, with examples from Kosovo, and privacy, which is especially pertinent today with terrorism turning into today’s communism, complete with our new form of McCarthyism. Of course, no alternative business book would be complete without pot shots at Professional Development Lit and traditional mass media. And it is with quotes on those two pernicious evil that I will close this review:

As early as the 1980′s, the French sociologist-philosopher Jean Baudrillard pointed out that the television viewer’s symbolic apotheosis as a receiver arrived when TV shows introduced canned laughter. He noted that television had reached a point at which the TV show was both the performer and its own audience, “leaving the viewer with nothing but pure amazement.” (p.107)