MinisterofDOOM wrote:Right. Surveys are a demonstrably s$%tty way of gathering information, because they're dependent upon too many external (not part of what's being measured) variables. Was the survey written well? Were the respondents reading properly?
CR's results are observably broken much of the time. Defending their methodolgy is like defending focus groups. Lots of people saying the same thing doesn't make it true. It makes them dumb.
Science is not a room full of survey respondents.
So by your logic, if 75% of a group of 20,000 2013 Ford Taurus owners say they experienced problems with their myfordtouch over the last year, if CR concludes Myfordtouch in the 2013 Ford Taurus is unreliable, it's probably untrue and they're dumb?
I don't understand what you mean by "broken". I guess I'm one of the few NICOnauts that subscribe to CR and have actually filled out their ownership surveys. If you were to examine their surveys, you would find their surveys are well written, their instructions are simple and cover many different areas of each car. And when you ask thousands of owners of the same make/model/year vehicle to identify problem areas they've encountered in the last 12 months via the same consistent format, it's rather easy to extrapolate which cars tend to hold up better than others and identify any weak areas.
Are there subscribers that do not answer the surveys truthfully? I'm sure there are a few that respond more positively or negatively. But given the sheer size of their surveys, (clearly not a focus group) you can still determine trends as it seems unlikely that enough subscribers would go to the effort to organize/collaborate with enough other subscribers in an evil conspiracy just to manipulate overall data maybe a tiny fraction. I've always answered their surveys truthfully and have never been approached to alter my responses. And unsurprisingly, for the cars I've owned, many of which I bought new and kept a long time, the CR owner reported problem areas for my model cars are more often than not similar to my own experiences.
I think one drawback to owner surveys, especially for older vehicles, is that so many have changed hands, many several times. That means folks could be experiencing problems that were the result of poor maintenance/abuse by a previous owner (in addition to themselves) and not the result of a poor design/build quality. But the survey is meant to be simply a source of insight, not gospel. And for what it is, I think CR does a reasonably good job of that despite your disdain for it.