Restoring symmetry, and another survey debunked
Feb 14, 2012
Reader John G. submitted a chart plus its improvement. Thank you!
The problem chart is used to present a "net promoter score" analysis by ABB (link). Net promoter score is the difference between people who will recommend a product or company and people who won't. The chart presents the components, the number of people who gave "red cards" and the number giving "green cards".
Unfortunately, the symmetry in the definition of the net promotor score is destroyed by this stacked bar chart. The red bars are all aligned against the vertical axis but the green bars aren't so it's difficult to compare their lengths.
John fixed this problem by aligning both sets of bars against a vertical axis. Sensibly, he places the red bars along the negative direction. He also orders the categories by "margin of victory", whichin effect is the net promoter score, with the category needing the most attention at the top.
The improved chart points out some of the complicating factors in understanding a metric that is composed of two components each of which vary and can be missing. For example, a category like "technical support" is rated among the highest overall but this conceals the fact that it is one category that receives many red cards. Also, consider "product/system quality" versus "health and safety": both categories end up with about the same net promoter score but the former category has many more respondents than the latter.
***
John also tried a scatter plot. This one requires some careful reading. The best categories are going to end up in the top left corner; it may be better to flip the red card axis to a descending order so that the top right corner is the best corner.
The diagonal rays (axes) are great visual aids to help figure out which categories are better and which are worst.
I am a little turned off by the crowdedness in the bottom left corner of the chart. Those are categories with relatively low levels of red or green cards, and also relatively balanced between reds and greens... that is to say, those are categories people don't care too much about, and among those people who care, there isn't a consensus about good or bad. In other words, the survey revealed very little of use about those categories. It bothers me that much of the survey ended up collecting data on such items.
I often find it worthwhile to plot such things (data sets which crowd the bottom left corner) on log scales. This spreads out the data points, and turns the radiating diagonals into parallel diagonals. The danger is that it can freak out people who don't understand log scales.
Posted by: derek | Feb 15, 2012 at 02:48 AM
You say "In other words, the survey revealed very little of use about those categories" just after you say "those are categories people don't care too much about, and among those people who care, there isn't a consensus about good or bad."
The useful thing the survey tells is the latter - those cartegories don't matter. To me, that's a useful piece of information.
Posted by: Tom West | Feb 15, 2012 at 10:18 AM
+1 Tom West. The degree of engagement is worth knowing. I would suggest improving the scatter plot by sizing the data points to the votes cast in each one, which would emphasise the significant points. If you wanted to be funkier, you could even colour the ones that represent a statistically significant result.
Posted by: Alex | Feb 16, 2012 at 09:23 AM
Derek: Log scales should be used with great care. In this case, I prefer not to use log scales because it will exaggerate the least interesting points of the chart.
Tom: Point taken. If you dare click on the link that says "show user created categories", you'd see why I think none of that data is meaningful, the way this survey was designed.
Posted by: Kaiser | Feb 16, 2012 at 11:51 PM
Both charts would benefit by focusing on the re-formulation of {#RED, #GREEN} -> {#RED+#GREEN, #GREEN-#RED} -- essentially a typical sum-difference chart.
The bar chart originally showed just the sums, and the re-formulation works much better as it highlights the differences. However it seems to have randomized the categorical dimension. the original chart sorted by #RED, but I would suggest that sorting by the sum or the difference makes more sense.
The scatterplot screams out for a sum-difference chart -- it makes the orientation correct and makes the h/v directions meaningful.
A log scale is not a great plan as counts are rarely exponential in nature. Counts often benefit from a square root transform (see intermediate statistical texts for reasons why) so that might be attempted if the sum-difference chart does not fix the issue
Posted by: Graham Wills | Feb 17, 2012 at 10:32 AM
What software did reader John G. use for his charts?
Posted by: John | Feb 18, 2012 at 08:47 AM
That one scatterplot label should be revised to say "Top 3/Bottom 17 vote count," not "Top 4." One of the strengths of the scatterplot (sum-difference would do this too) is that it highlights three separate clusters of vote counts: the 10 categories people don't care too much about (<100 total votes), the 7 they kind of care about (150 to about 200) and the 3 they care tons about (from about 300 and up).
I'd be tempted to scrap the blue and yellow lines separating the vote count clusters (they add to the chart's confusion of criss-crossing lines at first glance), and see what it would look to just circle the clusters, or maybe colour-code them. Sum-difference would give more options for highlighting the different clusters visually too.
Cool charts as always - and cool that John G. sent in improvements.
Posted by: Canadian Nathan | Feb 18, 2012 at 10:51 AM
John: It's Excel as far as I can tell. Obviously, he's a power user.
Posted by: Kaiser | Feb 21, 2012 at 07:45 PM
Graham: Excellent suggestions, the charts would be better, thank you.
John and Kaiser: Yes, I did use Excel 2003. Is your use of the term 'power user' genuine, or is it tongue-in-cheek?
Canadian Nation: it actualy is 4 and 17: there are 21 categories, and the fourth one above the line is just barely on the uphill side, hugging the 50/50 line: 'Product/System Quality'
Everyone: I love the comments, the more I learn the better I'll get!
Posted by: John G | Mar 01, 2012 at 03:59 PM
John G - it would be interesting if you also added a dot marker to show the actual net margin of victory. Can you post your file somewhere so that I could do this, plus have a play with the data?
Posted by: Jeff Weir | Mar 13, 2012 at 09:18 PM
"He also orders the categories by "margin of victory", whichin effect is the net promoter score, with the category needing the most attention at the top."
Well actually that has nothing to do with the Net Promoter Score...For more details go to www.netpromoter.com.
Secondly, when you are talking about customer loyalty the net card picture is not necessarily relevant. Technical Support is a case in point. Although there are many more green cards than there are red, there are still a significant number of customers that are dissatified, so John G's chart puts it way down on the lsit of priorities whereas in my book it would certainly warrant more attention.
Thirdly your point about the scatter chart and those issues that customers do not comment on much says just that...that these are categories that are not important to the customer, thereby allowing the company carrying out the survey to focus on those issues that are being apid more attention to by customers.
My view is that John G hasn't really understood what he is looking at.
Posted by: Michel B | Jun 16, 2013 at 03:59 PM
Haven't seen this post earlier, but I worked on the system this chart comes from. Few remarks:
- The categories are ordered this way because business people wanted it this way - the area to focus on was the one with most negative comments.
- The chart doesn't have much to do with NPS, but the first thing a surveyed person does is answering the NPS question and choosing score on a 0-10 scale. So it measures NPS (see the link, it lists NPS score as 23%)
It also allows (but does not require) respondents to mark some categories as "good" or "bad" (green/red) and provide comments. This allowed the mentioned chart to be created, but also some additional analysis - e.g. estimating what would the NPS score be for various categories (I don't remember the details anymore).
Posted by: mabn | Oct 18, 2014 at 06:29 PM