brownelearning

I finally took the time to code up a JavaScript that makes factor analysis tables look good. If you use SPSS to run such analyses, this page should probably help you out.

A grad student emailed me about the TICS. Specifically, the issue was how it compares to the Levels of Technology Implementation (LoTi) survey. Here are some of the things I came up with:

1. LoTi is a commercial survey, which has its advantages, but it is also not as open to scrutiny – or adaptation – as the TICS.

2. The TICS is aligned to the NETS-T while LoTi has its own framework.

3. Going by LoTi’s site, it’s clear that it is not in conformance with the latest AERA, APA, NCME Standards for Educational and Psychological Testing (which very few instruments are).

4. The LoTi concerns implementation of technology while the TICS measures self-efficacy regarding technology integration. Self-efficacy is mentioned twice on the LoTi website, once in an article that doesn’t mention LoTi (Eastin & Rose, 2000), and in another article (Meorsch, 1995) that hypothesizes self-efficacy’s relationship with LoTi (but no data is gathered).

Given these differences, it may be that the TICS is better suited for teacher education environments where…
1) the free and adaptable nature of the TICS is desirable,
2) the alignment to NCATE-accepted national standards is needed,
3) the academic rigor of the AERA, APA, NCME standards would be understood, and
4) the predictive nature of self-efficacy is advantageous.

The LoTi may be better suited for in-practice contexts where…
1) the LoTi’s cost is a small issue and its corporate backing is seen as legitimatizing,
2) accreditation isn’t an issue, so alignment to the NETS-T is less important,
3) conformance to research standards is not an concern, and
4) the concern is about actual practice, not potential practice.

Because I’m struggling to get my head around it all.

1. The Wiki for Evaluation and Assessment: I got fed up with textbooks, so my students are writing their own this semester. Teams of students have been assigned two chapters to write and provided access to my personal library of assessment texts. They’re expected to post a finished draft for each chapter the week before we cover it in class. I’m writing the first three chapters. The first chapter on reliability they read (and critiqued to shreds) this week; it needs severe revision. The second and third chapters (correlation and validity) are in the works and will be done this weekend.

2. Completing a review of the video analysis literature. We’re trying to use standardized video experiences to replace a small portion of field experience. We hope to overcome the paradox of using non-standardized field experiences to fulfill standards. We’ve been blocked by those who fear the state would never allow it, but I just learned that the new state commissioner of education, David Stiener, has a background in video analysis of teachers. Also, I’ll be meeting with Michael Preston of Teachers College at TEDxNYED in March. I need to get something together because my abstract was accepted to EDMEDIA Toronto in June.

3. I’m on the College Technology Council, which has sent representatives (including myself) to form a working group to decide what to do about our CMS going away. (It was bought out by Satan, I mean, Blackboard.) I’m on a committee for that working group that is assessing how faculty have used the CMS in the past (to form criteria for the next CMS), and a sub-committee that is looking into emerging technologies. Yes, it goes Council→Working Group→Committee→Sub-Committee.

4. I am now the chair of the unit assessment committee. That’s the group that design the data collection for all of the college’s teacher education accreditation needs. We’re currently developing a more robust system to assess the graduate-level teacher education outcomes.

5. I’ve completed an analysis of foreign language enrollments based on government databases. I know it was wrong, but I did it without completing a lit review. Now I have to go back and finish that before I can submit the paper.

6. I’m conducting a lit review of language students’ motivation for another paper based on six years of national surveys submitted by Middle East language learners.

7. SUNY Brockport and CUNY Cortland won a Title Vi grant to increase globalization instruction at the GE level. They’re including a dual-site Chinese course, so I promised to hook them up with the efforts in Utah (BYU-Utah Hindi, and BYU-UVU Chinese). I also owe the PI an email explaining my concerns with their rather arbitrary comparison level on two groups of students…

8. The SSRC wants to continue our relationship, which is great, but I’m very busy. I’ll respond to their email this weekend when I get a moment to breathe.

9. A dear friend has asked me to check out his work on Brain Honey. That should be interesting.

10. Oh, and somewhere in there I’m teaching three preps this semester. We’re three weeks into the semester and one course doesn’t even have a completed syllabus yet!

11. Post next week’s EDI 419 activities for enterprising students who want to take advantage of their week off.

12. I almost forgot: I need to write an evaluation protocol for a massive grant application the NMELRC is turning in.

A factor analysis is a statistical method for reducing data into fewer variables. Imagine that I had a personality questionnaire with 50 items about your likes, dislikes, habits, beliefs, and attitudes. If I gathered responses from enough people, I could put all their responses into a factor analysis and discover that many of those items are inter-related. For example, every item that concerned health would be grouped into one factor.

There are two things I appreciate about factor analysis: First, it’s how often the results reveal new understandings. Sure, you could probably pick out the items that will cluster before you gather the data, but there is usually a surprise or two – an item within a different factor, a factor that you didn’t predict, etc. Second, factor analysis results in a table of item-factor loadings. This table displays the degree to which each item “loads” on each factor.

Once you have this table, you can look at the item loadings and inductively name each factor. In our personality questionnaire, we could see that every item regarding interpersonal relationships loaded on one factor, so we could name that factor “People-orientation” or something.

Despite how useful factor analysis is at bringing order to chaos, I’ve never been comfortable sharing the process with less-statistically-oriented colleagues. It has been my experience that eyes glaze over at the site of such huge tables of numbers. My problem was that I needed to present the results of just such an analysis to my department, and I knew many of them had very little statistics experience.

Luckily, I was trolling some blogs and I noticed a tag cloud. “Hmmmm,” I thought. A few lines of PHP later, I had this:

Each bracketed string is a description (or in some cases the entire text) of an item from our student teacher evaluation form. The size of the text reflects the item’s loading on that factor. Any item that loads above a .400 is in green; items loading between -.400 and .400 are gray; and items with loadings less than -.400 are red. The items are also ordered from greatest loading to least.

Though I still had to explain a few times how it worked, I think we went a lot further with a word cloud than we would have gone with just a data table.

Inside Higher Ed has an article on the session my SSRC group held at the 50th Anniversary of Title VI conference last week. I’m not mentioned, but that suits me just fine: If a quantitative analysis depends on who completes it, something is wrong.

This started out as a response to another blog post, but it got too long, so I spun it off here.

I’ve often wondered about how well qualitative methods are generally applied in instructional design research. My specific training is in quantitative methods, so I’m not qualified to make that judgment, but I have observed a certain incongruence between the amount of rigor required by my colleagues in other fields, and the little effort many instructional design researchers exert.

For example, I came across the term autoethnographic in a study by Kupczynski et al (2008). The authors only cite one reference supporting the method, and it’s a textbook. I asked an anthropology friend of mine (a professor at American University) who specialized in qualitative methods about it and she about fell over. Her exact words were, “Good night, what is research coming too?!” She has warned me previously against data that is “too clean,” and shown how to include rigorous checks for trustworthiness (which I would call validity).

Compare this to a good friend of mine (and a great researcher) who passed his dissertation defense with a few revisions. One of those revisions caught my attention. To paraphrase his committee chair:

“In your methods section you described a negative case analysis [which is a method used to verify trustworthiness], but you didn’t actually do one. So, before we approve your dissertation, you’ll need to strike that section from your methods.”

I think it may be that instruction designers are practitioners first, and researchers second. During an open discussion at PIDT two years ago, a professor claimed that one of the things holding up instructional design as a field was the reluctance of journals to accept qualitative methods. He specifically mentioned that ETR&D had rejected several good submissions from him and his students. Cliff Mims from the University of Memphis spoke up because he is (was?) the editor over qualitative studies for that journal. He countered that he read a lot of submissions, but could usually tell within the first two pages that the “qualitative study” wasn’t rigorous enough to be published.

During the next break, I approached Cliff and asked him what his qualifications were to judge the quality of qualitative research. He responded with a litany of certificates he held and workshops he had attended on real qualitative methods. His comments jibed completely with what I’ve experienced working with sociologists and other qualitative experts at the Social Science Research Council and the National Academies.

It was there that I envisioned a project that has languished in its nascent stages for two.. almost three years: I want to construct a rubric based on qualitative practices from anthropology and sociology, and then score 200 randomly selected qualitative theses and dissertations in instruction design. Even if the great majority of them are aligned with the guidelines for rigor, this study would identify areas of relative weakness with the field.

Cliff loved the idea at PIDT, but I just haven’t had the time to follow through with it. So, if any graduates student reading this are looking for a good thesis or dissertation project, feel free to take this one. Just cite me.

I’m reviewing the proof of an article I have coming out in Computers in the Schools. The article summarizes work that is well over two years old, so it’s not surprising that I have forgotten a lot of what I wrote. But a certain, almost thrown-away point hit me pretty hard this time.

The article concerns the initial development of the Technology Integration Confidence Scale, including the collection of rudimentary evidence to support the validity of its intended interpretations. One piece of evidence entailed educational professionals categorizing the scale’s items into one of the six standards to which the items were written. In other words, I wrote each item to address a specific standard, so I asked other people to categorize the items into the standards. If these other people categorized the items into the categories I had intended, the evidence would support the item-standard connection. The results were mixed:

As shown in Table 3, 9 of the 22 items were classified in their a priori NETS-T by less than 80% (4 of 5) of the raters. However, the raters’ relation to technology integration apparently affected their classifications. The administrator only placed 8 items in their a priori categories, while the two in-practice teachers did so with 16 and 18 items. It is possible that educators who fill different positions might interpret and apply the NETS-T differently, but such a question is beyond the scope of this study. (Emph. added.)

Such an issue may not have interested me then, but now I am profoundly interested. Could it be that the contexts in which administrations and teachers exist affects how they interpret national standards? A conclusive study into this would have profound implications for education.

As part of my continuing work with the Social Science Research Council, I will be attending two meetings in New York City next week. It would be pointless for me to fly out twice within a week, so I’m staying the entire week and hoping to catch up on a backlog of work. Specifically,

1. Submit the final TICS v3 analysis to SITE.
2. Catch up on grading and post undergraduate midterm grades.
3. Crack open a new FOIA-requested version of EELIAS IRIS.
4. Read a two opposing books on standardized testing.
5. Clean out my inbox.

A quick update to explain that I have neglected my blog to assemble my renewal dossier. I’m using MediaWiki as a CMS to catalog why my contract should be renewed with the State University of New York. Take a look and let me know if you have any suggestions.

Once that’s done, I head off to NYC for a week to consult. I have several blog posts in the hopper, including my reaction to reviewing 15 AERA submission. I’ll post those when I get a chance.

Using video analysis software in teacher training is not new. In fact, several researchers have recently demonstrated its potential (see references). However, all of the research to date has viewed video analysis as something added to the typical teacher training curriculum. None have attempted to integrate it into the required preservice field experience.

Read the rest of this entry »