Comments on Turning in his grave 1TypePad2008-05-07T03:19:00Zjunkchartshttps://junkcharts.typepad.com/junk_charts/tag:typepad.com,2003:https://junkcharts.typepad.com/junk_charts/2008/05/turning-in-his/comments/atom.xml/derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef011168ce9fc8970c2009-03-09T19:32:56Z2009-03-09T19:32:56Zderekhttp://i-ocean.blogspot.com/Jon, that's great. I initially, and for months afterward, didn't follow up Nick's fine and generous code, but when I...<p>Jon, that's great. I initially, and for months afterward, didn't follow up Nick's fine and generous code, but when I finally did, I liked it a lot. I look forward to seeing what you've done with it. </p>Jon Peltier commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef011168cd99f0970c2009-03-09T11:05:05Z2009-03-09T11:05:05ZJon Peltierhttp://peltiertech.com/WordPress/I've made some refinements to Nick's LOESS function: LOESS Smoothing in Excel<p>I've made some refinements to Nick's LOESS function:</p>
<p><a href="http://peltiertech.com/WordPress/loess-smoothing-in-excel/" rel="nofollow">LOESS Smoothing in Excel</a></p>derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef01053590e231970c2008-10-16T18:03:01Z2008-10-16T18:03:04Zderekhttp://i-ocean.blogspot.com/Oh, sorry, I didn't spot that bit. I thought you had written it local, but not weighted.<p>Oh, sorry, I didn't spot that bit. I thought you had written it local, but not weighted. </p>nick commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef01053590b591970c2008-10-16T16:40:09Z2008-10-16T16:40:09Znickof course, if you wanted to use all the data points, you could either a) set l equal to the...<p>of course, if you wanted to use all the data points, you could either a) set l equal to the number of data points or b) remove the "Do Until d.Count = l" loop from the VBA code. If you wanted a different weighting function than the tri-cube that I used, replace <br />
"w.Add i, (1-(d.Item(i)/maxDist)^3)^3"<br />
with whatever weighting funciton you prefer.</p>nick commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef01053588b519970b2008-10-16T16:34:29Z2008-10-16T16:34:29ZnickSorry its not what you were looking for. When I was writing it, I was working off of information from...<p>Sorry its not what you were looking for. When I was writing it, I was working off of information from <br />
http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd144.htm</p>
<p>and used the example from:<br />
http://www.itl.nist.gov/div898/handbook/pmd/section1/dep/dep144.htm</p>
<p>which my code matches exactly. The local points (l=7 in the example page) ARE weighted using the tri-cube weighting function (see in my code the "w" dictionary). </p>
<p>Although the NIST page mentions "At each point in the data set a low-degree polynomial is fit to a subset of the data..." from the example, they were using a 1st degree (ie linear) polynomial, so I also wrote my code to do a weighted linear regression on local points.</p>derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef010535907be9970c2008-10-16T15:19:51Z2008-10-16T15:19:52ZderekAh, it uses a restricted number of points, each with 100% weighting. I can implement that without VBA. What I'm...<p>Ah, it uses a restricted number of points, each with 100% weighting. I can implement that without VBA. </p>
<p>What I'm looking for is a way to take all the points, but weight them according to a function, e.g. Gaussian, of distance from the newX point. I've always been hazy about the difference between Cleveland's cute names, loess and lowess; is that the difference? </p>
<p>Thanks for the reply. </p>nick commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef010535901f8a970c2008-10-16T13:16:22Z2008-10-16T13:16:22ZnickProbably not the most elegant implementation, but it works. Its a UDF that takes in as arguments the known X's...<p>Probably not the most elegant implementation, but it works. Its a UDF that takes in as arguments the known X's (xRng), known Y's (yRng), number of points to use in the regression (l), and the point to calculate (newX). </p>
<p>Function loess(xRng As Range, yRng As Range, l As Integer, newX)<br />
Dim x As Dictionary<br />
Set x = CreateObject("Scripting.Dictionary")<br />
Dim y As Dictionary<br />
Set y = CreateObject("Scripting.Dictionary")<br />
Dim d As Dictionary<br />
Set d = CreateObject("Scripting.Dictionary")<br />
Dim w As Dictionary<br />
Set w = CreateObject("Scripting.Dictionary")</p>
<p>'// convert ranges to dictionaries and fill in distances<br />
For i = 1 To xRng.Count<br />
x.Add i, xRng.Item(i)<br />
y.Add i, yRng.Item(i)<br />
d.Add i, Abs(xRng.Item(i) - newX)<br />
Next i</p>
<p>'// loop, removing largest distance until we're down to the number of points we want<br />
Do Until d.Count = l<br />
maxDist = -1<br />
For Each i In d.Keys<br />
If d.Item(i) > maxDist Then<br />
maxDist = d.Item(i)<br />
mx = i<br />
End If<br />
Next i<br />
d.Remove mx<br />
Loop</p>
<p>'// Find max distance, then calc weights (w) using scaled distances<br />
maxDist = -1<br />
For Each i In d.Items<br />
If i > maxDist Then maxDist = i<br />
Next i<br />
For Each i In d.Keys<br />
w.Add i, (1 - (d.Item(i) / maxDist) ^ 3) ^ 3<br />
Next i</p>
<p>'// now to calculate I to V and denom for linear regression<br />
For Each i In w.Keys<br />
one = one + w.Item(i)<br />
two = two + x.Item(i) * w.Item(i)<br />
three = three + x.Item(i) ^ 2 * w.Item(i)<br />
four = four + y.Item(i) * w.Item(i)<br />
five = five + x.Item(i) * y.Item(i) * w.Item(i)<br />
Next i<br />
denom = one * three - two ^ 2</p>
<p>'// calculate slope (slp), intercept(inter), and finally the loess value<br />
slp = (one * five - two * four) / denom<br />
inter = (three * four - two * five) / denom<br />
loess = slp * newX + inter</p>
<p>'// empty the dictionaries<br />
Set w = Nothing<br />
Set d = Nothing<br />
Set y = Nothing<br />
Set x = Nothing<br />
End Function</p>derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef01053587d609970b2008-10-16T06:51:11Z2008-10-16T06:51:11Zderekhttp://i-ocean.blogspot.com/nick, that's great to hear, and something I've long wanted to do myself. Is it in the form of a...<p>nick, that's great to hear, and something I've long wanted to do myself. Is it in the form of a UDF with a format similar to FORECAST(), or some other method? And would you be willing to share it? </p>nick commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef0105358c8ea8970c2008-10-15T16:25:52Z2008-10-15T16:25:53ZnickI have implemented loess in vba from scratch.<p>I have implemented loess in vba from scratch.</p>Jon Peltier commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5529d1dca88342008-05-28T02:19:06Z2008-05-28T02:19:06ZJon Peltierhttp://peltiertech.com/WordPress/Kelly - I don't know. Dunkin used to be pretty good, but they've worried so much about their menu selections...<p>Kelly -</p>
<p>I don't know. Dunkin used to be pretty good, but they've worried so much about their menu selections that they've forgotten how to make coffee. I can't drink it any more.</p>
<p>I never used to like Starbucks coffee either, but I find it doesn't seem so burnt the last few times I've tried it. It's a little pricey, but if you stick to regular coffee without any embellishments, it's not so bad.</p>Mike commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e55278d51d88332008-05-23T16:14:41Z2008-05-23T16:14:42ZMikeI drank starbucks once. Once. It is burnt, disgusting and way over priced. I can purchase more than enough Folgers...<p>I drank starbucks once. Once. It is burnt, disgusting and way over priced. I can purchase more than enough Folgers for me than that garbage they sell. Folgers, Dunkin, whatever - anything is better than starbucks. And if Obama and his followers like it, then all the more reason to wonder about starbucks.</p>D Kelly O'Day commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522f1add88342008-05-08T16:37:25Z2008-05-08T16:37:25ZD Kelly O'Dayhttp://processtrends.comLowess (aka Loess) is a locally weighted regression technique link. Lowess uses all the data by fitting a series of...<p>Lowess (aka Loess) is a locally weighted regression technique<br />
<a href="http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd144.htm" rel="nofollow"> link</a>. Lowess uses all the data by fitting a series of "local" regressions, giving insight into the overall relationship which may vary over the range of x. </p>
<p>Rather than eliminate data, lowess actually uses all the data several times by computing weighted regressions in multiple subsets of the data.</p>
<p><br />
</p>Kaiser commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522ecc0588342008-05-08T13:18:02Z2008-05-08T13:18:03ZKaiserhttp://junkcharts.typepad.comThanks Derek. Both the original exercise and what I did fall into the realm of exploratory data analysis. The linear...<p>Thanks Derek. Both the original exercise and what I did fall into the realm of exploratory data analysis. The linear fit was poor and I tried to show why it was so. In general, though, Michael's point is valid.</p>derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522eb13c88342008-05-08T11:36:23Z2008-05-08T11:36:23ZderekIt's okay to remove data temporarily, to show how sensitive the line is to the data. The data that "best...<p>It's okay to remove data temporarily, to show how sensitive the line is to the data. The data that "best fits" the theory, in this case, only fits the ends. The bulk of the data doesn't back the theory up at all. </p>
<p>This is why I mentioned lowess. It's basically a fancy mathematical way of doing exactly what Kaiser did: see what happens to the line when you take the ends away. If the linear regression slope goes away, you know the line wasn't strong enough to bear the weight of the hypothesis laid on it. </p>
<p>The lowess version would have a wavy pattern that would be much less impressive as proof of a trend than a linear least-squares fit, which is the next closest thing to a simple mean (a mean is an aggregate measure boiling the data down to one number, a line boils it down to two numbers). </p>Michael commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522df3e488342008-05-08T02:09:29Z2008-05-08T02:09:29ZMichaelDoes removing all the data that best fits the theory disprove the theory? Or is this simply a case of...<p>Does removing all the data that best fits the theory disprove the theory?</p>
<p>Or is this simply a case of cherry picking - only choosing the data that agrees with your theory and discarding that which doesn't.</p>Ethan commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522ce60d88342008-05-07T17:28:47Z2008-05-07T17:28:48ZEthanhttp://www.urbanspoon.com(whoops -- that county-level data is for Aler)<p>(whoops -- that county-level data is for Aler)</p>Ethan commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e552154d6688332008-05-07T17:27:26Z2008-05-07T17:27:27ZEthanhttp://www.urbanspoon.comProfessor, I have missed you. Thanks for providing a more rigorous analysis! Indeed, the correlation is as weak as bad...<p>Professor, I have missed you. Thanks for providing a more rigorous analysis! Indeed, the correlation is as weak as bad coffee. I'll try harder on my next statistical foray.</p>
<p>Oh, and Vozome -- you can find county level data, albeit not well organized, here:<br />
http://www.cnn.com/ELECTION/2004/pages/results/president/</p>Aler commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e552152ca688332008-05-07T15:52:00Z2008-05-07T15:52:01ZAlerI'd like to do a little analysis of my own on data like this. For example, how does the Obama-Clinton...<p>I'd like to do a little analysis of my own on data like this. For example, how does the Obama-Clinton vote depend on the rural-urban ratio of a state (or the average population density), or the democrat-republican vote in the 2004 elections?</p>
<p>Can anyone recommend a good source for the raw data? I'd rather not type everything in by hand if I can avoid it. State by state would be fun, but county by county would be amazing.</p>vozome commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5521517c688332008-05-07T15:07:45Z2008-05-07T15:07:45Zvozomehttp://www.oecd.orgabout dual-scaled axes and proportional scales... while I agree with Jon Peltier's example sometimes the opposite happens: same scale for...<p>about dual-scaled axes and proportional scales... while I agree with Jon Peltier's example sometimes the opposite happens: same scale for two variables that have nothing to do with each other, for instance indices or percentages that happen to go for 0 to 100. <br />
like in your (Kaiser's) example they are often misused to invoke correlation when there is none. </p>
<p><br />
</p>Matt commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522c726588342008-05-07T14:35:15Z2008-05-07T14:35:15ZMattOne problem is that true elitists, wouldn't drink at starbucks if they have a local roaster. Starbucks is pretty mainstream,...<p>One problem is that true elitists, wouldn't drink at starbucks if they have a local roaster. Starbucks is pretty mainstream, I prefer my local house and I'm basically a snob. </p>Kelly O'Day commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e55214f2fb88332008-05-07T13:35:02Z2008-05-07T13:35:02ZKelly O'Dayhttp://processtrends.comDerek - After trying to get Excel - VBA to handle lowess and panel charts for quite a while, I've...<p>Derek - After trying to get Excel - VBA to handle lowess and panel charts for quite a while, I've decided to move up to r. </p>
<p>There is a great Excel add-in for r, <a href="http://www.unt.edu/benchmarks/archives/2004/may04/rss.htm" rel="nofollow">link to tutorial</a> that gives Excel users full access to r graphics. To me, this is the best of both worlds, Excel for data manipulation, r for statistical and graphical analysis.</p>
<p>Kaiser - great post. I look forward to seeing more from Mr. Urbanspoon. </p>Jon Peltier commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e55214b62788332008-05-07T11:29:31Z2008-05-07T11:29:31ZJon Peltierhttp://peltiertech.com/WordPress/Speaking of secondary axes, Stephen Few discussed them in "Dual-Scaled Axes in Graphs-Are They Ever the Best Solution?" (the answer...<p>Speaking of secondary axes, Stephen Few discussed them in "<a href="http://www.perceptualedge.com/articles/visual_business_intelligence/dual-scaled_axes.pdf" rel="nofollow">Dual-Scaled Axes in Graphs-Are They Ever the Best Solution?</a>" (the answer is no), and I countered with a class of examples where they are acceptable in "<a href="http://peltiertech.com/WordPress/2008/03/26/secondary-axes-that-work-proportional-scales/" rel="nofollow">Secondary Axes that Work - Proportional Scales</a>"</p>derek commented on 'Turning in his grave 1'tag:typepad.com,2003:6a00d8341e992c53ef00e5522b72fa88342008-05-07T05:32:16Z2008-05-07T05:32:16ZderekSpeaking of regression, has anyone ever implemented lowess (locally-weighted regression), in VBA or otherwise, for Excel? I see Kelly O'Day...<p>Speaking of regression, has anyone ever implemented lowess (locally-weighted regression), in VBA or otherwise, for Excel? I see Kelly O'Day has a note that he's looked and come up blank. </p>