brown(e)learning

Here’s a direct link to the CHum 250 Syllabus: http://syllabus.byu.edu/view/D3MbN1KvBr0h.html

CHum 250 Survey: https://byu.qualtrics.com/SE/?SID=SV_5jUFJL3RM976YPa

The week’s session of Understanding Educational Research included the great online primer to research design by Trochim. Before diagramming the various designs in articles we had found, we reviewed the classic models:

The controlled, randomized experiment:
R O X O
R O   O

The quasi-experiment:
O X O
O   O

The longitudinal study:
O O O O

And, of course, the non-experiment:
X O

As we discussed the strengths and weaknesses of each model, we tried to think of research purposes for which each design was suited. Of course, only one of these model can by itself establish causality, which is why it is the “gold standard” of educational research. Both quasi-experiments and longitudinal studies are useful for establishing trends and formulating hypotheses (which are then tested with the first model), but what about that last model?

The non-experiment is pretty much useless. Sure, you give some treatment and you gather some data, but one of two fatal flaws lingers: 1. Either the point of the study was to reveal something about the treatment, but, without a pretest, we can’t say anything about it; or 2. the treatment wasn’t the focus of the study, in which case it shouldn’t be part of the design.

Towards the end of our discussion, one of the literacy masters students shouted, “The treatment is irrelevant!” I thought that was so astute that I added it to the slide with an APA-formatted citation to the student’s comment.

The students, most of whom are in-practice teachers, saw this two slides later:

A study that claimed to establish the effect of any treatment without at least a pre-test would have a hard time getting published in an academic journal. Yet that is the model we’ve been using for ten years to determine whether teachers, schools, districts, and states are effective.

Some states have moved to the “growth model” for teacher evaluations, which would look like this:
O X O

The major improvement with this model is that we can now establish the change that occurred during the treatment period. On the other hand, it does not tell us the degree to which the treatment was responsible for the observed change. This model is still not appropriate if its results are to be used to hold individuals and groups accountable.

It is becoming sadly commonplace to see policymakers misuse social science methods, but, in this case, there is also a high degree of hypocrisy: At the same time state offices of education have attempted to discover teacher effectiveness via research models that cannot produce such evidence, those same offices have called for teachers to use only “evidence-based methods” of instruction. It’s a classic case of do-as-we-say-not-as-we-do.

I’ve been growing more concerned with the field of educational research, especially in testing and assessment. It seems that nothing the experts say and no research they do is able to convince policymakers that they’re misusing tests and test results.

For example, it’s been six years since Freakonomics showed how to detect cheating on state tests. It’s worth noting that the authors thought a modicum of test security measures would make most anti-cheating analyses unnecessary. You know, an ounce of prevention…

Despite the popularity of that book and the soundness of its arguments, very few states have adopted anti-cheating techniques. It took the Atlanta Journal Constitution‘s application of those methods to uncover an enormous cheating program in Georgia (as well as similar scandals in Pennsylvania, Connecticut, and New Jersey) to get states to move.

So it was some relief that I read this announcement from NYSSBA. Some highlights:

1. Teachers will no longer proctor tests to their own students.
2. Teachers will no longer be able to grade their own students’ tests.
3. The county-based canning/scoring systems will be replaced with one centralized system.

If you’re like I was when I arrived in NY, your first reaction was, “Wait, teachers, whose evaluations depend in part on their students’ test scores, are the ones who give and score their students’ tests?” Yeah, crazy, huh?

I guess I should be happy that some change is finally happening, but, in reality, I’m trying to figure out how to cause a scandal that will force states to develop tests that yield adequately reliable results.

My good friend Margaret Merrill sent this article from the New York Times, which, once again, calls into question the value of integrating technology into education.

In a nutshell: schools are spending billions on technology, even as they cut budgets and lay off teachers, with little proof that this approach is improving basic learning.

This conundrum calls into question one of the most significant contemporary educational movements.

This should come as no surprise to researchers in educational technology, a field that has never felt the need to justify itself through, say, quantitative data.

“The data is pretty weak. It’s very difficult when we’re pressed to come up with convincing data,” said Tom Vander Ark, the former executive director for education at the Bill and Melinda Gates Foundation and an investor in educational technology companies. When it comes to showing results, he said, “We better put up or shut up.”

I don’t hope that they suddenly find the data because that data should have been there all along. It would take a decade’s worth of research to justify the last decade of investment in educational technology. I only hope the pendulum doesn’t swing back with too much force.

I also hope other fields are watching this tragedy play out. I can’t help thinking of the emerging subfield of “Digital Humanities,” which is essentially the integration of technology into the traditional aspects of humanist inquiry (another field that has never seen the need to justify itself). The digital humanities is the spitting image of educational technology circa 1995. Though the technology they employ is more sophisticated than what was (and is) used in education, digital humanists are battling the same cultural issues that educational technologists thought they defeated more than a decade ago.

Digital humanists are making the same mistakes as well.

If the digital humanities continues to exist mainly in an echo chamber, you can bet that they will be rejected by their own. And even if they somehow gain entrance into the humanist canon of research methods (e.g., via administrative fiat), within ten years they will see the same counter-technology movement going on in education today. That is, unless they move forward deliberately and document the benefits of their methods.

But the price of this documentation and the motivation for this deliberation is the confession that technology may not be the solution or that its benefits may not outweigh its costs. Digital humanists would have to be suspicious of their methods – of their field – and that is difficult for any academic. On the other hand, such sincere research ought to be the hallmark of academics.

I don’t blog every time someone contacts be about the TICS, but today I received an email with an intriguing question:

I’m going to be working with a school where the teachers are going to be given iPads. I was wondering if you had given any thought to adding items [to the TICS] that measured the confidence of the use of mobile technologies?

The quick answer is no, but the more sincere (and correct) answer is maybe.

On the one hand, I don’t see how using a tablet or other wireless device changes effective technology integration practice. On the other hand, the TICS items should represent a broad range of current classroom technologies, which would include tablets. On the other hand (yes, we’re up to three hands), the TICS will never be an effective measure if it shimmies and shakes to the rhythm of each month’s gadgetry. There are no TICS items on podcasts, location-based social networking, SmartBoards or SmartTables, ARGs, cloud-based computing… Why would mobile solutions be included?

I know there are many, many, many researchers in educational technology working with tablets and other mobile technologies. Their work is already important and may prove even more so. However, given the time it takes to craft, pilot, and validate self-efficacy items, I don’t think mobile technologies deserve a spot on the TICS.

But don’t fret! This is not the end of the road. The TICS is distributed under the Creative Commons By/Share-Alike license. This means that you can alter the TICS to your heart’s content, so long as you 1) credit the creator of the original TICS and 2) allow others to modify the resulting product. I encourage you to add your own items on mobile technologies (or whatever your specific tool-of-interest may be), gather data on those items, and publish your findings (even if it’s on your blog or in an email to me).

Two months ago I set up a page on my domain that contained an SQL injection vulnerability. I had accidentally signed up for a service that scans my server for security issues, and I intended to tests its effectiveness. Well, that service finally ran a scan, and here are the results:

The scan picked up a XSS flaw on the page because the PHP echoed the username GET var, which could have contained malicious Javascript. The Firefox add-on NoScript successfully blocked my attempts to exploit it. They get points for that one.

But they completely missed the more severe SQL injection.

I’m tempted to mention the service by name, but I have a feeling that would violate the TOS agreement somehow. Instead, I recommend staying away from all such site-scanning scams.

There was a web meeting today on instructional design. I was in another meeting, so I was unable to tune in, but I was interested in something from the session’s abstract:

Author and designer Victor Papanek (1985) defines design as “the conscious and intuitive effort to impose meaningful order” (p.4) This can mean that whether we are involved in instructional design, or in other endeavors in which something is to be created different than what existed before; “all that we do, almost all the time, is design, for design is basic to all human activity” (Papanek, 1985, pg. 4).

Beginning with Papanek’s specific definition, I can’t get over the union of “conscious and intuitive effort.” My dictionary places these adjectives in slight opposition: Conscious entails awareness, while intuition occurs in the absence of reason. Perhaps these terms were included as either-or descriptors (e.g., “both conscious and intuitive efforts may be counted as design”). Such inclusion would be consistent with another point I’ll make below.

I’m less certain the requirement that design assumes the intention to “impose meaningful order.” That limitation appears useful to distinguish design from its superset art, but I waiver on ostracizing creation without meaning. I’d like to ponder the implications of this specification a bit longer.

I noticed that fields attempting to define themselves often do so with implicit goals of inclusion. This motivation drives the theorists toward definitions that are no less abstract than the term’s generally accepted meanings. Take, for example, Papanek’s comment that design is “all that we do, almost all the time.” To quote David Merrill’s protestations against learning objects, “If something is everything, then it is nothing.” In other words, defining something to encompass anything or almost everything renders the term useless; it would be more useful to define what design isn’t.

Perhaps the difference between Papanek’s definitions and my own efforts to lock-down the meaning of design has been the angle of attack: Papanek was a designer who believes his field to be truly broad, while I was a design student trying to delineate the difference between my field and art. I eventually accepted this distinction:

Design is creation within constraints.
Art is creation that may or may not be constrained.

By my definitions, some design is clearly a subset of art, but art is a larger umbrella. I placed the “constraints” in design because I noticed that in practical fields – from my father’s engineering, to my graphic design (and later to my software, research, and instructional design) – the design process included acknowledgement of the functional requirements and limitations of the work.

I’m not sure that either of these definitions is adequate, but hearing Papanek’s definition stirred the memories of a long-settled self-debate.

I attended a regional conference on instruction and technology this week. I’ll admit I didn’t know what to expect – I had never been to anything smaller than a national conference – but I should have been able to predict that the sessions would be highly variable in their quality. Of course, there were the vendor demonstrations and the how-to reports devoid of data, but there were also more discussion-oriented presentations on issues of common-yet-unexplored techniques, and business reports on local issues.

I was satisfied with the quality of several of the presentations I attended, especially one on rubrics wherein, once the presenter discovered (by – gasp! – asking the audience questions) that most of the attendees had experience developing rubrics, she decided to let us do the talking. She would pose a question (which she knew to be the topic of her next slide), and then she would moderate our conversation. After four or five comments, she would flip to the next slide and show us that we had covered everything on it. Her effort in organizing the information wasn’t wasted just because the group in the room already knew it; see, we only knew it collectively, and she helped move that information from collective to individual.

Sadly, the worst hour I spent was during on keynote address. I’ve known for a while that the educational technology field is driven more by marketing hype (and the artificial demand it creates) than by the effectiveness of the technology. This presentation made me wonder the degree to which historical revisionism has helped maintain that unhealthy producer-consumer relationship.

The presenter stated as fact that the early computer-assisted educational programs translated the production mentality of early computers into a one-size-fits-all approach to education. While there were mass-produced products of this type, it’s wrong to ignore the more instructionally-appropriate efforts that included student tracking and automatically adjusted to their needs.

Ten years ago I sat on a committee with some very bright professors in the field. We discussed requirements for instructional authoring systems and, as each “cutting edge” feature was brought up, someone would ask a distinguished professor, “Didn’t [insert thirty-year-old system] have that?” He would nod or give a simple, “yep” in response.

The ed tech field’s collective ignorance of the depth and meaningfulness of their roots is not only wrong because it slights the visionaries of decades past; it’s also wrong because it allows us to be wooed by technologies that appear new, but are only new in that industry has never pushed them before.

This ignorance is most observable in our field’s unabashed embrace of the mobile “app” model. Those of us who have given decades of effort to establishing open, standards-based technology thoroughfares should be revolted that making our wares available on monopolized toll-roads is now seen as a requirement. (For more reading, see The App is Crap.)

The weather has finally turned here in Western NY, so I’m biking to work more. I finally setup my Macbook so I can access its desktop from my home Windows PCs. Of course, I need this connection to be encrypted, so I had to tunnel it over SSH (which means using Vine Server rather than Apple’s Screen Sharing). There are numerous tutorials for doing this, but this one held to more important information: the exact PuTTY tunnel configuration.

So, more for my records than for your information, here are my screenshots.