Archive for July, 2010

A Quick Way to Make Sense of Factor Analysis Results

Projects
 Posted by jeremy on July 16th, 2010

I finally took the time to code up a JavaScript that makes factor analysis tables look good. If you use SPSS to run such analyses, this page should probably help you out.

Posted in Projects | No Comments »

Supporting VIAR: The Potential Threats Approach

Teaching
 Posted by jeremy on July 2nd, 2010

I could have done a better job explaining to my summer students how to establish VIAR. So I’m going to post here a quick worksheet to organize validation efforts:

1. State how you the assessment results will be used. Assessment results are often used for more than one purpose, state each separately.

2. Each use requires that the assessment results be interpreted in some way. In other words, each use implies that the results mean something. What is it? (It is possible that uses will share interpretations.)

3. For each interpretation, think of two or three “threats” – conditions that if true, would lessen the correctness of that interpretation (i.e. reduce the VIAR).

4. Categorize each of these VIAR threats into the following categories (adapted from APA, AERA, NCME, 1999):

Threats in the content of the assessment
  • Threats in how the respondents (examinees, students, etc.) interact with the assessment
  • Threat in the assessment results’ relationship to other data (including a lack of a relationship)
  • Threats in the assessment results (unexpected patterns of responses, or a lack of expected patterns in the responses).

    • Several of your threats may occupy the same category. (See Notes 1 and 2.)

    5. For each category of threat that is represented, describe evidence you could gather that would indicate the degree to which those threats are real. These description could be surveys (formal and informal), experiments, interviews, analyses of the assessment results, etc. (See Note 5.)

    6. Simulate gathering the evidence. Use this random number generator to simulate gathering the evidence.

    7. State the action you will take to address any threat that was not dismissed by the evdience.

    Note 1: You’ll notice I omitted the category “Consequences of Testing,” which is listed in the 1999 Standards. I did this because the consequences may have an impact on a too-broadly defined concept of validity, but it has no place in VIAR.

    Note 2: I’m not crazy about Step 4. It does help organize the threats, but two threats in the same category would still require different evidence. I think that step may be me forcing the APA et al. model on this problem.

    Note 3: Step 5 would be a great place to reference the scientific method, null hypothesis, etc.

    Posted in Teaching | Comments Off