No, a broken clock is not reliable
There’s a very common – and very inaccurate – analogy about reliability that spreads like a cancer. Though it is easy to understand, the metaphor conveys several misconceptions about reliability, and it’s worth debunking whenever you hear or read it.
First, a little background. The terms reliability and validity are old and used in several contexts, each of which adds distinct nuances. Most scholars learn the concepts in regards to research methods (sorry philosophers), and it is from that field that this horrible analogy originates.
A broken clock is reliable because reliability is consistency of measure. A broken clock always gives you the same reading; it’s very consistent. But a broken clock would only be valid twice a day: when the actual time happens to be the time displayed on the broken clock.
I won’t go into validity today, I think I’ve beat that drum to death, but I will use this flawed analogy to demonstrate why reliability can only refer to a specific type of consistency. My one-line response to this analogy is…
So a working and calibrated clock wouldn’t be reliable because it’s inconsistent? I mean, every time I look at it, it gives me a different reading.
The consistency in reliability refers to the consistency in measuring a variable. In context of the analogy, the variable is time and a clock that doesn’t change as the time changes does not consistently measure the target variable. Twice a day, when the clock accurately reflects the actual time, the measure would be accurate (has no error), but five minutes later it’s less accurate (has some error), and six hours later it would be completely inaccurate (have extreme error). Because of this inconsistency in regards to its measure of time, a broken clock cannot be deemed reliable.
The case of the broken clock also reveals an important, but often overlooked detail about reliability: It’s not the instrument (the clock) that is more-or-less reliable, but the results/readings/information that the instrument produces. It would be more appropriate to say that the time displayed on a broken clock is not reliable.
If anyone asks, “but what about test information in item response theory?” that’s going to be my next post. Let’s see if I can paint myself out of that corner.