Information gain and loss

The previous two posts indicated that CNN, TWC and Intellicast had the best on-line weather forecasting accuracy by looking at the median and mean error in predicting daily low and high temperatures over 41 days.  Is it possible to differentiate between those three?

For that, we need more data so I switched from summary statistics back to the data.  In this new chart, the day by day errors were plotted.  The gridlines labelled errors within 5 degrees, which is an arbitrary guideline for acceptable / unacceptable.  The three scatters looked remarkably similar although CNN appeared to hit the bull's eye (the middle square) with less bias (errors more evenly distributed) but not much better accuracy overall (similar number of unacceptable errors).

Comments

[derek]

Making the symbols for the data points "C", "I", and "T" might have been a good idea when they all shared the same graph space, but I can't see the point when they have separate spaces in a small multiple. I'd rather see a title on each graph in the series.

"I" and "T" were particularly unfortunate choices because when, as I do, I scanned the graph before looking at the text to see what was being presented, I thought the I was an error bar with no terminal dashes, and the T was an error bar with a terminal dash on the top.

[失踪]

I agree with derek. The small multiples are better compared using dots instead of letters. By visual inspection I cannot determine if the higher density of Cs is just a felt higher density because a C takes more space than a I. So I end up trying to count the letters to be sure.. Better use dots.

[Hadley]

You might want to have a look at the lattice or ggplot packages for making it easier to do small multiple plots in R. (I'm the author of ggplot, so I'm biased as to which is better)

[TQ]

"...errors within 5 degrees, which is an arbitrary guideline for acceptable / unacceptable."

The NWS uses +/- 3°F error in its verification statistics.

It/s a given there will be errors in temperature forecasts. What/s left to decide is how much error is tolerable. NWS uses 3°F. The present analysis uses 5°F.

I interpret the data plots as follows:

LOW T forecast...
CNN: 8 too warm; 6 too cold
Intel: 8 too warm; 5 too cold
TWC: 5 too warm: 4 too cold

TWC had fewer _extreme_ errors; therefore...TWC was more accurate forecasting the LOW temperature.

HIGH T forecast...
CNN: 2 too warm; 2 too cold
Intel: 2 too warm; 1 too cold
TWC: 2 too warm: 2 too cold

All sources showed equal skill forecasting the HIGH temperature.

[Kaiser]

TQ, thanks for the counts. Those are measuring the variance, and it shows that using +/-5, the three were statistically the same.

What I found interesting was the lower bias shown by CNN. Errors typically could be due to bias or variance. Here, the bias is revealed in the evenness of the spread around (0,0).