When using psychological tests, it is important to ascertain if they have good validity (i.e. test the concepts they claim to test). Obviously, if a test has already undergone evaluation to show that it has good validity (e.g. it correlates well with other measures assessing the same concept) then this can be a great time saver for you. However, I've recently been struck by a few measures which, while having previously undergone testing to examine validity, seem a priori to either be muddled or downright contradictory. (No doubt more seasoned psychologists than me can think of other examples).
A while back I completed an online survey whereby one was asked to place oneself in an imaginary scenario (stage 1). One is asked to imagine oneself committing a crime (let's say robbing a bank). In this story one is requested to imagine that one feels a sense of excitement and enjoyment in committing this crime. After having read (and presumably daydreamed about) this scenario, there are a series of follow-up questions (stage 2). One of these questions asks you to indicate how much one would enjoy robbing the bank (with the option of saying "not at all"). But you have already been told to imagine that you do enjoy it at stage 1. The question can't really be probing how much the respondent would actually enjoy such an scenario, as the survey as a whole is telling the person that they would enjoy it in this imaginary world. On mentioning this to a researcher using these questions he pointed out that he had previously noticed this shortcoming and was working on re-validating the questions-it annoyed me that he was left doing this as the original designers had not taken more care to ensure that the questions made sense in light of the scenarios that went with them.
A second example is perhaps less irritating, as the question that annoyed me was not necessarily contradictory but rather a bit muddled. However, the response to my griping from the author who used it was telling. An example question they mentioned was a triple-barrel question, let's say along the lines of "In order to enhance economic growth, our nation should cut corporation tax, reduce workers' rights and abolish the minimum wage". When I pointed out that this was a triple-barrel question which didn't allow for people having differing views on the impact of minimum wage versus the impact of corporation tax on economic growth, the person using the scale pointed out that as the validity and reliability of the questionnaire had been assessed in that format, they were sticking with it.
Why rush into spending the time and effort of collecting a large set of data to ensure the validity of a measure when it only takes an intelligent person in a room to see whether or not the questions actually make sense and are clear-cut?
No comments:
Post a Comment