CILIP published the rebranding survey results, or a summary of them. What follows is a critique, and I acknowledge the help of others who've contributed analysis on social media, but the responsibility for this is mine and mine alone. There were two surveys, one on the rebrand, and a second on renaming.
General
- they don't say how long the surveys ran for, nor give the questions asked. We have no statement of aims, or the research question the survey was designed to answer
- for the first survey, they had 3,928 respondents, whom they divide into members, activists and non-members, but the first two categories are not mutually exclusive. And the percentages don't add up 68%+6%+21%=95%. What happened to the other 5%?
- for the second, on naming, they had 1,142 responses. We're not told how many were members or non-members
- it's unclear how multiple responses from the same respondent were prevented
- clearly there's been some rounding going on, but inconsistently, and sometimes incorrectly
The rating scales:
- the scales are fundamentally flawed; in scales of this sort, the middle option is neutral. Describing this as OK, which is too close to Well, is a serious error; a usual way to describe this middle option, number 3 on a 1-5 scale would be neither positive nor negative. Not only this, the reporters try to claim all the middle preferences as positives
- rating scales are in any case often criticised for their openness to biases, central tendency bias, acquiescence bias and social desirability bias
The second survey on naming
- a better and clearer way [if you wanted people to understand the survey, rather than use it to make propaganda] to present this data would have been to use weightings
- the presentation of the data my be appropriate for an ad-hoc meeting at Ridgmount Street, but is seriously inadequate as a basis for decision making
To quote one of those who looked at the data on Twitter, 'I just get the feeling it was thrown together, because it wasn't intended to be published, badly presented, misleading and even inaccurate'.
What we need now is the anonymised raw data. Will CILIP release it? I have asked.
So, taking the traditional critical appraisal questions for a qualitative study, my answers are as follows:
Was the research design appropriate to address the aims of the research? Not as it stands; a survey was a reasonable approach, but not this one
Was the recruitment strategy appropriate to the aims of the research? No idea, it isn't stated.
Were the data collected in a way that addressed the research issue? Again, can't tell, we know very little about how the data were collected
Has the relationship between researcher and participants been adequately considered? We have no indication whatsoever that this was thought about. We have only the sketchiest information about the researchers, to the effect that they are a from by the name of Spencer Dubois
Have ethical issues been taken into consideration? We don't know. At least the survey is anonymised, and we can guess that those who took part gave informed consent
Was the data analysis sufficiently rigorous? Vide supra
Is there a clear statement of the findings? Absolutely not
How valuable is the research? Not at all; there's no context given, no way the reader can asses what, if anything, the results mean
Recent Comments