You can follow this conversation by subscribing to the comment feed for this post.
God, I can't even understand what they're trying to say here. Is there somewhere where we can get access to the original data set? This is incomprehensible.
I agree with Zachary. 1) Does Autism Spectrum include Autism? 2) Why are there no labels on Unwin's chart? I have no idea how to correlate the two charts. What do the green/grey vs. black/white portions represent? 3) Do the three circles in the Venn diagrams represent different diagnoses or different diagnostic tools? 4) What does the parenthetical percentage indicate?
It's impossible to see how to best represent data before you understand the meaning.
Assuming the data can be explained, one approach is to see if the Venn diagrams can truly be made to represent reality. That is, if it is possible to adjust the circle sizes and their overlap so that proportions are correct.
For example, in a two-circle Venn, if there is 50% overlap between the two data sets, then the two circles should be positioned so the intersection represents 50% of the total area taken up by the circles. Similarly, if one data set is smaller than the other, then the circle representing that data set should be proportionally smaller.
If the data cannot work this way, then the Venn diagram is not the best solution. (It may still work at an abstract level requiring mental interpretation, but not as a visual map to accurately depict quantity relationships.)
No Autism Spectrum does not include Autism. They are different conditions. The three circles represent three different way of assessing the condition, one is through interview with parents, another is an interview with a trained clinician and another one is pre-linguistic test.
The first number is the number of kids taking the test at age 2. The parenthesis indicates the percentage of same kids with that condition (autism or asd) at age 9. This is a very unique experiment as there are not that many studies looking at difference in diagnostic over time.
As I mentioned in Andrew Gelman's blog, either PL-ADOS or ADI-R are no better than a flip of a coin when taken at age 2 (age 3 is a different story). The specially trained clinician input provides the needed element for a good diagnosis.
Some of my original comments can be found here:
http://nuit-blanche.blogspot.com/2007/09/on-difficulty-of-autism-diagnosis-can.html
Any other charts that you guys can come up with, please forward to Andrew Gelman.
Zachary, this graph is not an answer to the challenge, but I hope it helps you understand what's going on. 172 children were tested at age two using three different methods, then their diagnosis five years later was noted (assuming diagnosis to have been well-confirmed by then). 135 of the children (78% of the original 172) were found to suffer from some form of autism spectrum disorder, and of those, 100 cases (58%) were full autism.
A stacked bar chart shows the individual techniques well enough, but what about combinations of the two, or all three together? The Venn diagram gives all the numbers (all three techniques in agreement on a "yes" are right in 90% of predicted cases of full autism, and 96% if predicting some form of autism spectrum disorder), but it's not easy to read. Is there a better way to show all the combos? A matrix graph perhaps, or a Sankey diagram? Even possibly the dreaded doughnut graph?
OK, here's my version of the chart. It is not perfect (!) but definitely makes certain aspects more understandable.
I felt the most important aspect was how well the age 2 assessment agreed with the age 9 assessment. Shading shows this; the darker the area, the stronger the assessment.
Clearly, two tests are better than one, and three tests are even better. Also, the tests seem generally of equal predictive power except perhaps for the Clinician Diagnosis of autism, which was 100% accurate (although there were only 2 data points here...).
The small numbers so prevalent in the original were removed. This is because one of the numbers (percentage) is now expressed via shading, and the I felt the other number (count of kids who took the test) was not germane to the Main Point of the chart. IMHO, a separate table with this data would be fine for the number crunchers.
Unfortunately, the chart caption does have to be read to understand what is going on. But this is an improvement over the original, which required going to Igor's Nuit Blanche blog (grin!) for any hope of understanding.
If I was to improve my version of the chart further, I would need to ask additional questions to better understand the study's methods, goals and results.
You've made a mistake in your interpretation: all the tests are given in all 172 cases. The area outside all three circles is "all tests say No", not "no test given".
I revised my horizontal bar chart by adding context of group size here, on Andrew Gelmans' blog. I'm not sure I completely understand the study, but I believe the format is better than a venn diagram.
Derek wrote:
"You've made a mistake in your interpretation: all the tests are given in all 172 cases. The area outside all three circles [42 cases or 14%] is 'all tests say No', not 'no test given'."
Patrick replies:
I based my label of "No test given" on something Igor wrote in the comments on his Nuit Blanche blog. He was discussing the "42 (14%)" label outside the left set of circles. Igor wrote that "as far as I can tell ... 42 kids were identified as needing an assessment at age 2 but did not do any of the three tests, yet 14% had a diagnosis of autism at age 9." (italics added)
In addition, I don't think the area outside the circles is "all tests No". If the left set of circles (Autism) had an overall prevalence of 58%, that should mean that 42% of people tested did not have Autism -- e.g., "all tests No". But the area outside the left circles is labeled as "14%".
Of course, I may well be wrong since the original chart and data are so confusing. It will be interesting if we can find out what the real answer is (gg!).
OK, back again with a new version that has many benefits.
-- It shows the exact numbers of children in each group, while keeping the Venn relationships of the groups.
-- There is now a legend for the shading that shows the exact value of the percentage agreement.
-- The 42 children who did not take the test are better identified (I hope this is correct!)
Note that I only did the "left half" of the original chart, because placing and counting the dots was so tedious. I leave the similar right half as an exercise for interested/obsessed readers like myself!
I based my label of "No test given" on something Igor wrote
All that means is that Igor is as wrong as you are. If you look at the original paper, it says:
"One hundred ninety-two children were prospectively studied from the time they were referred for evaluation for possible autism [...] The 172 children with data at both ages 2 and 9 years form the basis of this report"
and
"Three measures of diagnosis were obtained at ages 2 and 5 years."
All three measures were obtained at age two for all 172 children, and the yes/no is about whether the diagnosis was positive or negative. If "No" meant "no measure obtained", then where are the numbers for "measure obtained but diagnosis=No"?
If the left set of circles (Autism) had an overall prevalence of 58%, that should mean that 42% of people tested did not have Autism -- e.g., "all tests No". But the area outside the left circles is labeled as "14%".
The 58% you refer to says that out of all 172 children, 100 were diagnosed autistic by age nine. The 14% says that out of 42 children diagnosed as other than autistic at age two by all three measures, 6 were nevertheless diagnosed as autistic by age nine.
Note that where all three measures said "Yes", the numbers were 51 at age two, of which 46 turned out autistic by age nine, or 90%. In other words, where all three measures said "No", they were wrong 14% of the time, and where they all said "Yes", they were wrong 10% if the time.
(This is all assuming the age nine best estimate is "right")
Here is my attempt at this. Based on information provided by Igor my feeling is that the question behind the Venn Diagram is "which combinations of tests are consistent over time".
Oh, Igor is also incorrect about the numbers for Venn diagram B (Autism spectrum) not including the numbers for Venn diagram A (Autism). They do. The paper distinguishes Autism from other autism spectrum disorder diagnoses such as "Pervasive Developmental Disorder, not otherwise specified". B includes them all, while A includes only full autism.
The ultimate junk chart is one whose creator has not understood the data. No amount of expertise with the tricks of graph making can compensate for that.
The real failure here is that only the creator of this venn diagram, or those familiar with every detail of the data, understand the information without extensive explanation. I think the purpose of this discussion is to come up with a better method than a venn diagram for displaying this type of information. For instance, if the point is to show the effectiveness of including a clinician in the process, the venn fails miserably. Another major problem with the venn approach here is including two on the same page, and expecting the reader to make comparisons. I'm not saying that you can't figure that out, but it is not clear to the average reader.
Maybe you should list your conclusions from your review of all of the detail, and illustrate the best way to communicate that information, venn diagram or not.
Derek, thank you for reading the original report and clarifying the data. I agree that the creator of a chart has to understand the data. So I have made yet another version which based on my best current understanding of what you and others have written.
In this version, each child is indicated by a dot. The dot color shows whether the child had no autism, had autism found at age 2 but not at age 9, or had autism found at both age 2 and age 9.
The Venn diagram is moved to the background and serves to organize each group of dots.
Note that the information in each group is encoded three ways: number of light dots vs. dark dots, percentage written out, and shading of the underlying Venn diagram.
I added a title that makes it clear what the chart is trying to communicate.
I did not do the right side of the original chart (Autism Spectrum) since it would be so similar to the left side -- which took a loooong time!
I sent my attempt to the Gelman blog, but they haven't approved it yet, so here it is, presenting the numbers as a pair of stacked tornado charts, one each for autism proper and autism spectrum disorder diagnoses. That one assumes an exclusive diagnosis, in other words, if it says the clinician and the interview agree, that means the observation dissents.
This one includes the unanimous diagnoses as well. In other words, it's as if the third measure were never taken at all. It's a little harder to see that the numbers are the same as the Venn diagram in this version, but on the other hand it lets you answer the question "how would the diagnoses differ if I had only used two of the three measures?"
Basically, I am concentrating on the hierarchical structure of the data and the percentages (which are the important info here, IMHO). I think the Venn diagram is not showing any interesting information at all.
Very clever; I like your simplification. If I could make a suggestion...
I think the key question the chart is addressing is: "which of the four categories is most accurate?"
The answer is hiding a bit because the four bars are different lengths. What if all four bars are set to equal length, so we are comparing the percentage of correct and incorrect diagnoses?
The absolute numbers could be encoded separately. For example, vertical height of the bar (higher = more children in that group). Or shading; darker = more children. Or even just putting the number at the end of the bars (example: "n=172").
I think you have gone deeper into the data than I did. The original chart did not show which combinations of tests gave negative diagnoses, so I assume you pulled that out of the paper.
Also a very nice data simplification & clarification.
Unfortunately, I found it somewhat difficult to interpret the tree map at the bottom of your chart. This was especially true in your second set of charts (with re-ordered vertical bars) because some tree map lines were crossing. IMHO, the Venn has an advantage in instantly showing the visual relationship between the three tests.
Therefore, I used a color-coded Venn in combination with your chart to create still another version. My goal was to try to make it easier to correlate a given bar with its appropriate test(s).
There are a few flaws with my version. The most obvious is that the colors will be lost if the chart is reproduced in black-and-white. I have an idea for this but leave it for the future if I get REALLY bored (grin!).
Also, for now I am not handling the 42 cases of negative diagnosis. I also have an idea for how to depict these -- a mix of your chart, my Venn colors and Derek's negative/positive bars -- but again I leave this for the future.
My main goal in this version was to implement a Venn-to-bar color correspondence and see if others find it easier to read than the tree map ... or if it is just me (gg!).
Hmmm, now that I have looked at your commentary once more, I understand your concept of re-ordering the vertical bars to give additional data insight.
Specifically, when you re-ordered the bars so the most accurate test results were on the right, the resulting tree map automatically put the three tests in order of importance. As you wrote (and I missed the first time), the tree map demonstrates that "[t]he clinician's assessment thus is the most important criterion, followed by ADI-R and then PL-AIDOS."
I still think the Venn is easier to grasp for a quick overview of the test situation. But I do agree that the tree map -- and especially the re-ordering -- provides greater data insight once it is understood. Thanks!
Patrick, I am arguing that the Venn diagram shows irrelevant information. It doesn't really matter all that much how many children got a positive result from which test, it's more about which combination of tests was actually correct. Of course the numbers are important to know how well supported the percentages are, but they're just not the focus of this study. At least that's my understanding of the data.
God, I can't even understand what they're trying to say here. Is there somewhere where we can get access to the original data set? This is incomprehensible.
Posted by: zachary wilson | Sep 28, 2007 at 09:45 AM
I agree with Zachary. 1) Does Autism Spectrum include Autism? 2) Why are there no labels on Unwin's chart? I have no idea how to correlate the two charts. What do the green/grey vs. black/white portions represent? 3) Do the three circles in the Venn diagrams represent different diagnoses or different diagnostic tools? 4) What does the parenthetical percentage indicate?
It's impossible to see how to best represent data before you understand the meaning.
Posted by: Chris Hibbert | Sep 28, 2007 at 01:14 PM
I agree that the data seems incomprehensible.
Assuming the data can be explained, one approach is to see if the Venn diagrams can truly be made to represent reality. That is, if it is possible to adjust the circle sizes and their overlap so that proportions are correct.
For example, in a two-circle Venn, if there is 50% overlap between the two data sets, then the two circles should be positioned so the intersection represents 50% of the total area taken up by the circles. Similarly, if one data set is smaller than the other, then the circle representing that data set should be proportionally smaller.
If the data cannot work this way, then the Venn diagram is not the best solution. (It may still work at an abstract level requiring mental interpretation, but not as a visual map to accurately depict quantity relationships.)
Posted by: Patrick Murphy | Sep 28, 2007 at 01:21 PM
my response can be found @
http://www.stat.columbia.edu/~cook/movabletype/mlm/
Posted by: Stack Lee | Sep 28, 2007 at 01:25 PM
Chris,
No Autism Spectrum does not include Autism. They are different conditions. The three circles represent three different way of assessing the condition, one is through interview with parents, another is an interview with a trained clinician and another one is pre-linguistic test.
The first number is the number of kids taking the test at age 2. The parenthesis indicates the percentage of same kids with that condition (autism or asd) at age 9. This is a very unique experiment as there are not that many studies looking at difference in diagnostic over time.
As I mentioned in Andrew Gelman's blog, either PL-ADOS or ADI-R are no better than a flip of a coin when taken at age 2 (age 3 is a different story). The specially trained clinician input provides the needed element for a good diagnosis.
Some of my original comments can be found here:
http://nuit-blanche.blogspot.com/2007/09/on-difficulty-of-autism-diagnosis-can.html
Any other charts that you guys can come up with, please forward to Andrew Gelman.
Thanks,
Igor.
Posted by: Igor Carron | Sep 28, 2007 at 03:12 PM
Zachary, this graph is not an answer to the challenge, but I hope it helps you understand what's going on. 172 children were tested at age two using three different methods, then their diagnosis five years later was noted (assuming diagnosis to have been well-confirmed by then). 135 of the children (78% of the original 172) were found to suffer from some form of autism spectrum disorder, and of those, 100 cases (58%) were full autism.
A stacked bar chart shows the individual techniques well enough, but what about combinations of the two, or all three together? The Venn diagram gives all the numbers (all three techniques in agreement on a "yes" are right in 90% of predicted cases of full autism, and 96% if predicting some form of autism spectrum disorder), but it's not easy to read. Is there a better way to show all the combos? A matrix graph perhaps, or a Sankey diagram? Even possibly the dreaded doughnut graph?
Posted by: derek | Sep 28, 2007 at 03:33 PM
OK, here's my version of the chart. It is not perfect (!) but definitely makes certain aspects more understandable.
I felt the most important aspect was how well the age 2 assessment agreed with the age 9 assessment. Shading shows this; the darker the area, the stronger the assessment.
Clearly, two tests are better than one, and three tests are even better. Also, the tests seem generally of equal predictive power except perhaps for the Clinician Diagnosis of autism, which was 100% accurate (although there were only 2 data points here...).
The small numbers so prevalent in the original were removed. This is because one of the numbers (percentage) is now expressed via shading, and the I felt the other number (count of kids who took the test) was not germane to the Main Point of the chart. IMHO, a separate table with this data would be fine for the number crunchers.
Unfortunately, the chart caption does have to be read to understand what is going on. But this is an improvement over the original, which required going to Igor's Nuit Blanche blog (grin!) for any hope of understanding.
If I was to improve my version of the chart further, I would need to ask additional questions to better understand the study's methods, goals and results.
Posted by: Patrick Murphy | Sep 28, 2007 at 06:19 PM
You've made a mistake in your interpretation: all the tests are given in all 172 cases. The area outside all three circles is "all tests say No", not "no test given".
Posted by: derek | Sep 28, 2007 at 07:08 PM
I revised my horizontal bar chart by adding context of group size here, on Andrew Gelmans' blog. I'm not sure I completely understand the study, but I believe the format is better than a venn diagram.
http://www.stat.columbia.edu/~cook/movabletype/mt-tb.cgi/1188
Posted by: Stack Lee | Sep 28, 2007 at 08:23 PM
Derek wrote:
"You've made a mistake in your interpretation: all the tests are given in all 172 cases. The area outside all three circles [42 cases or 14%] is 'all tests say No', not 'no test given'."
Patrick replies:
I based my label of "No test given" on something Igor wrote in the comments on his Nuit Blanche blog. He was discussing the "42 (14%)" label outside the left set of circles. Igor wrote that "as far as I can tell ... 42 kids were identified as needing an assessment at age 2 but did not do any of the three tests, yet 14% had a diagnosis of autism at age 9." (italics added)
In addition, I don't think the area outside the circles is "all tests No". If the left set of circles (Autism) had an overall prevalence of 58%, that should mean that 42% of people tested did not have Autism -- e.g., "all tests No". But the area outside the left circles is labeled as "14%".
Of course, I may well be wrong since the original chart and data are so confusing. It will be interesting if we can find out what the real answer is (gg!).
Posted by: Patrick Murphy | Sep 28, 2007 at 10:23 PM
OK, back again with a new version that has many benefits.
-- It shows the exact numbers of children in each group, while keeping the Venn relationships of the groups.
-- There is now a legend for the shading that shows the exact value of the percentage agreement.
-- The 42 children who did not take the test are better identified (I hope this is correct!)
Note that I only did the "left half" of the original chart, because placing and counting the dots was so tedious. I leave the similar right half as an exercise for interested/obsessed readers like myself!
Posted by: Patrick Murphy | Sep 29, 2007 at 01:23 AM
I based my label of "No test given" on something Igor wrote
All that means is that Igor is as wrong as you are. If you look at the original paper, it says:
"One hundred ninety-two children were prospectively studied from the time they were referred for evaluation for possible autism [...] The 172 children with data at both ages 2 and 9 years form the basis of this report"
and
"Three measures of diagnosis were obtained at ages 2 and 5 years."
All three measures were obtained at age two for all 172 children, and the yes/no is about whether the diagnosis was positive or negative. If "No" meant "no measure obtained", then where are the numbers for "measure obtained but diagnosis=No"?
Posted by: derek | Sep 29, 2007 at 03:08 AM
If the left set of circles (Autism) had an overall prevalence of 58%, that should mean that 42% of people tested did not have Autism -- e.g., "all tests No". But the area outside the left circles is labeled as "14%".
The 58% you refer to says that out of all 172 children, 100 were diagnosed autistic by age nine. The 14% says that out of 42 children diagnosed as other than autistic at age two by all three measures, 6 were nevertheless diagnosed as autistic by age nine.
Note that where all three measures said "Yes", the numbers were 51 at age two, of which 46 turned out autistic by age nine, or 90%. In other words, where all three measures said "No", they were wrong 14% of the time, and where they all said "Yes", they were wrong 10% if the time.
(This is all assuming the age nine best estimate is "right")
Posted by: derek | Sep 29, 2007 at 03:12 AM
Here is my attempt at this. Based on information provided by Igor my feeling is that the question behind the Venn Diagram is "which combinations of tests are consistent over time".
My graph attempt is available here
Posted by: Bernard Lebelle | Sep 29, 2007 at 03:32 AM
Oh, Igor is also incorrect about the numbers for Venn diagram B (Autism spectrum) not including the numbers for Venn diagram A (Autism). They do. The paper distinguishes Autism from other autism spectrum disorder diagnoses such as "Pervasive Developmental Disorder, not otherwise specified". B includes them all, while A includes only full autism.
The ultimate junk chart is one whose creator has not understood the data. No amount of expertise with the tricks of graph making can compensate for that.
Posted by: derek | Sep 29, 2007 at 03:52 AM
Derek,
The real failure here is that only the creator of this venn diagram, or those familiar with every detail of the data, understand the information without extensive explanation. I think the purpose of this discussion is to come up with a better method than a venn diagram for displaying this type of information. For instance, if the point is to show the effectiveness of including a clinician in the process, the venn fails miserably. Another major problem with the venn approach here is including two on the same page, and expecting the reader to make comparisons. I'm not saying that you can't figure that out, but it is not clear to the average reader.
Maybe you should list your conclusions from your review of all of the detail, and illustrate the best way to communicate that information, venn diagram or not.
Posted by: Stack Lee | Sep 29, 2007 at 08:18 AM
Derek, thank you for reading the original report and clarifying the data. I agree that the creator of a chart has to understand the data. So I have made yet another version which based on my best current understanding of what you and others have written.
In this version, each child is indicated by a dot. The dot color shows whether the child had no autism, had autism found at age 2 but not at age 9, or had autism found at both age 2 and age 9.
The Venn diagram is moved to the background and serves to organize each group of dots.
Note that the information in each group is encoded three ways: number of light dots vs. dark dots, percentage written out, and shading of the underlying Venn diagram.
I added a title that makes it clear what the chart is trying to communicate.
I did not do the right side of the original chart (Autism Spectrum) since it would be so similar to the left side -- which took a loooong time!
Posted by: Patrick Murphy | Sep 29, 2007 at 04:56 PM
Patrick, I like that one a lot.
I sent my attempt to the Gelman blog, but they haven't approved it yet, so here it is, presenting the numbers as a pair of stacked tornado charts, one each for autism proper and autism spectrum disorder diagnoses. That one assumes an exclusive diagnosis, in other words, if it says the clinician and the interview agree, that means the observation dissents.
This one includes the unanimous diagnoses as well. In other words, it's as if the third measure were never taken at all. It's a little harder to see that the numbers are the same as the Venn diagram in this version, but on the other hand it lets you answer the question "how would the diagnoses differ if I had only used two of the three measures?"
Posted by: derek | Sep 29, 2007 at 05:26 PM
Patrick, Derek:
I encourage you both to post your graphs at perceptualedge.com under discussion / graph design / alternative to venn diagram.
Posted by: Stack Lee | Sep 29, 2007 at 06:32 PM
Hey, Kaiser. I'm jealous that you get so many comments on your site! Are you more popular, or is it the interactive way you asked the question?
Posted by: Andrew Gelman | Sep 29, 2007 at 08:10 PM
Great challenge! I have posted my own redesign on EagerEyes.
Basically, I am concentrating on the hierarchical structure of the data and the percentages (which are the important info here, IMHO). I think the Venn diagram is not showing any interesting information at all.
Posted by: Robert Kosara | Sep 29, 2007 at 10:09 PM
Derek --
Very clever; I like your simplification. If I could make a suggestion...
I think the key question the chart is addressing is: "which of the four categories is most accurate?"
The answer is hiding a bit because the four bars are different lengths. What if all four bars are set to equal length, so we are comparing the percentage of correct and incorrect diagnoses?
The absolute numbers could be encoded separately. For example, vertical height of the bar (higher = more children in that group). Or shading; darker = more children. Or even just putting the number at the end of the bars (example: "n=172").
I think you have gone deeper into the data than I did. The original chart did not show which combinations of tests gave negative diagnoses, so I assume you pulled that out of the paper.
Again, very nice data simplification!
Posted by: Patrick Murphy | Sep 29, 2007 at 10:16 PM
Robert Kosara --
Also a very nice data simplification & clarification.
Unfortunately, I found it somewhat difficult to interpret the tree map at the bottom of your chart. This was especially true in your second set of charts (with re-ordered vertical bars) because some tree map lines were crossing. IMHO, the Venn has an advantage in instantly showing the visual relationship between the three tests.
Therefore, I used a color-coded Venn in combination with your chart to create still another version. My goal was to try to make it easier to correlate a given bar with its appropriate test(s).
There are a few flaws with my version. The most obvious is that the colors will be lost if the chart is reproduced in black-and-white. I have an idea for this but leave it for the future if I get REALLY bored (grin!).
Also, for now I am not handling the 42 cases of negative diagnosis. I also have an idea for how to depict these -- a mix of your chart, my Venn colors and Derek's negative/positive bars -- but again I leave this for the future.
My main goal in this version was to implement a Venn-to-bar color correspondence and see if others find it easier to read than the tree map ... or if it is just me (gg!).
Thanks again for a very interesting approach.
Posted by: Patrick Murphy | Sep 30, 2007 at 12:09 AM
Robert Kosara --
Hmmm, now that I have looked at your commentary once more, I understand your concept of re-ordering the vertical bars to give additional data insight.
Specifically, when you re-ordered the bars so the most accurate test results were on the right, the resulting tree map automatically put the three tests in order of importance. As you wrote (and I missed the first time), the tree map demonstrates that "[t]he clinician's assessment thus is the most important criterion, followed by ADI-R and then PL-AIDOS."
I still think the Venn is easier to grasp for a quick overview of the test situation. But I do agree that the tree map -- and especially the re-ordering -- provides greater data insight once it is understood. Thanks!
Posted by: Patrick Murphy | Sep 30, 2007 at 12:28 AM
Patrick, I am arguing that the Venn diagram shows irrelevant information. It doesn't really matter all that much how many children got a positive result from which test, it's more about which combination of tests was actually correct. Of course the numbers are important to know how well supported the percentages are, but they're just not the focus of this study. At least that's my understanding of the data.
Posted by: Robert Kosara | Sep 30, 2007 at 01:26 AM