See what I mean

Andrew Sullivan said that the Rasmussen poll has been disconcordant with other polls in recent months, and he shows us the graphs (from Pollster.com) to show us.  A good example of effective visualization.

Note what makes it work: identical vertical scales on both charts, identical time frames, matched colors (disapproving red, approving black).

Reference: "Rasmussen vs. the rest", Andrew Sullivan blog, Dec 29 2009.

[Update - 1/5/2010: A few constructive comments, including a stern note from "A Professor", sent me scrambling to see if I have been too trusting of the experts.  Thankfully, the original source of these charts, Pollster.com, provides interactive tools that can be used to test the suggestions. 

It is true that the Gallup poll is a counter-balance to the Rasmussen, biased in the opposite direction, although when one looks at the evidence below, one still has to conclude that the variation between the Rasmussen and all others is much more striking than that between the Gallup and all others.  In particular: the disapproval proportion has exceeded the approval since August in the Rasmussen when this pattern is still not completely clear in the aggregate of all other polls by December.  (The cross-over appears to be inevitable unless some new policy sways public opinion back to Obama soon.)

On the balance, I'd still consider Sullivan's point to be valid.  Then again, I don't consider myself an experts on polls so there could well be other anomalies hiding within the dozens of polls.  I am a bit intrigued/disturbed by the fact that the Gallup apparently did not measure disapproval until August, or perhaps there was a glitch in the plotting software.

Of course, any polls or market research interviews can be easily manipulated, via selection of samples, via using leading questions, via the structure of questionnaires, etc. etc.  That's why the Pollster-style charts showing us the aggregate trends are crucial to look at.

Cherry-picking is to be frowned upon but sometimes the cherry-picked item is indeed an outlier, and at other times it is not.  When an entire group is being taken out, and the underlying dataset is large, as in here, the risk of falsely throwing out good data is smaller but it is always good to be vigilant.

On this last point, I'm again grateful to our vigilant readers for pointing out problems with the initial post.]

Comments

[Bob]

Nice to see how Sullivan is so selective in his picking of "outliers". He always mentions Rasmussen, yet he never seems to mention polls that are obvious outliers when they benefit Obama. The AP poll of a week and a half ago is a perfect example. It shows Obama with a 56% approval rating during a time period when every other poll had him in the upper 40s. This president hasn't had 56% approval in months and months.

[Alan Abramowitz]

You're missing the point, Bob. Rasmussen is not only consistently an outlier but it has a very disproportionate influence on the overall averages because of it is releasing new results all the time. Rasmussen is in fact the only poll that has such an impact due to the combination of its status as an outlier and its frequency.

[Joel]

As a lapsed psephologist I write from the heart when saying that polls are at risk of being spun in many different ways, and fear that the comments section on this post may reflect it! However, Alan's right: we need to distinguish between an outlier and a trend. AP may be an outlier, but Rasmussen is a trend.

Without looking at the comparative question wording, if I were with Rasmussen I'd be rather worried.

[Tom Brady]

Alan,

What you don't say, however, is that Gallup plays a similar role, but in the opposite direction, since it consistently has higher approval ratings for Obama - their latest has his approval/disapproval at 49.6-43.4. Drop Gallup and the composite poll suddenly reads 50.9 DISapproval versus 46.2 approval. Not quite the impact of Rassmussen, but significant nonetheless.

In any event, I think Bob is right: Sullivan is being slightly disingenuous here.

Aside: By truncating the left hand y-axis, Sullivan also creates a visual perception suggesting a bigger difference between Rasmussen and the composite poll. It's defensible I guess, but....

[A Professor]

Dear Mr JunkCharts,

As a longtime regular, who recommends this blog to his statistics students, how could you miss the fact that Sullivan clearly cherry-picked his polls?

You lose credibility!

A Professor

[Kaiser]

All: thanks for the thoughtful comments. I did some more research and augmented the original post.

[A Professor]

Thanks so much for addressing the cherry picking issue. My mind is at ease!

A Professor

[otcconan]

Note that the Rasmussen only shows automated polls. People tend to be more honest if they are afraid of saying something that could be construed as embarrassing or, heaven forbid, racist.

[web developer]

I think that Rasmussen is the only poll that has an impact when it comes to the combination of its status as an outlier and its frequency.