Storm story, a masterpiece

The visual story published by the New York Times on hurricane Irma is a masterpiece. See the presentation here.

The story starts with the standard presentation of the trajectories of past hurricane on a map:


Maps are great at conveying location and direction but much is lost in this rendering - wind speeds, time, strength, energy, to name but a few.

The Times then switches to other chart forms to convey some of the other data. A line chart is used to convey the strength of wind speeds as the storms shake through the Atlantic. Some kind of approximation is used to straighten the trajectories along an east-west orientation.


The key insight here is how strong Irma was pretty far out in the Atlantic. The lines in the background can be brought to live by clicking on them. This view omits some details - the passage of time is ignored, and location has been reduced to one dimension.

The display then switches again, and this time it shows time and wind speed.


This shows Irma's strength, sustaining Category 5 level windss for three days. This line chart ignores location completely.

Finally, a composite metric called cyclone energy is introduced.


This chart also ignores location. It does show Irma as a special storm. The storm that has reached the maximum energy by far is Ivan. Will Irma beat that standard? I am not so sure.

Each chart form has limitations. The use of multiple charts helps convey a story from multiple perspectives. A very nice example indeed.


Sorting out what's meaningful and what's not

A few weeks ago, the New York Times Upshot team published a set of charts exploring the relationship between school quality, home prices and commute times in different regions of the country. The following is the chart for the New York/New Jersey region. (The article and complete data visualization is here.)


This chart is primarily a scatter plot of home prices against school quality, which is represented by average test scores. The designer wants to explore the decision to live in the so-called central city versus the decision to live in the suburbs, hence the centering of the chart about New York City. Further, the colors of the dots represent the average commute times, which are divided into two broad categories (under/over 30 minutes). The dots also have different sizes, which I presume measures the populations of each district (but there is no legend for this).

This data visualization has generated some negative reviews, and so has the underlying analysis. In a related post on the sister blog, I discuss the underlying statistical issues. For this post, I focus on the data visualization.


One positive about this chart is the designer has a very focused question in mind - the choice between living in the central city or living in the suburbs. The line scatter has the effect of highlighting this particular question.

Boy, those lines are puzzling.

Each line connects New York City to a specific school district. The slope of the line is, nominally, the trade-off between home price and school quality. The slope is the change in home prices for each unit shift in school quality. But these lines don't really measure that tradeoff because the slopes span too wide a range.

The average person should have a relatively fixed home-price-to-school-quality trade-off. If we could estimate this average trade-off, it should be represented by a single slope (with a small cone of error around it). The wide range of slopes actually undermines this chart, as it demonstrates that there are many other variables that factor into the decision. Other factors are causing the average trade-off coefficient to vary so widely.


The line scatter is confusing for a different reason. It reminds readers of a flight route map. For example:


The first instinct may be to interpret the locations on the home-price-school-quality plot as geographical. Such misinterpretation is reinforced by the third factor being commute time.

Additionally, on an interactive chart, it is typical to hide the data labels behind mouseovers or clicks. I like the fact that the designer identifies some interesting locales by name without requiring a click. However, one slight oversight is the absence of data labels for NYC. There is nothing to click on to reveal the commute/population/etc. data for central cities.


In the sister blog post, I mentioned another difficulty - most of the neighborhoods are situated to the right and below New York City, challenging the notion of a "trade-off" between home price and school quality. It appears as if most people can spend less on housing and also send kids to better schools by moving out of NYC.

In the New York region, commute times may be the stronger factor relative to school quality. Perhaps families chose NYC because they value shorter commute times more than better school quality. Or, perhaps the improvement in school quality is not sufficient to overcome the negative of a much longer commute. The effect of commute times is hard to discern on the scatter plot as it is coded into the colors.


A more subtle issue can be seen when comparing San Francisco and Boston regions:


One key insight is that San Francisco homes are on average twice as expensive as Boston homes. Also, the variability of home prices is much higher in San Francisco. By using the same vertical scale on both charts, the designer makes this insight clear.

But what about the horizontal scale? There isn't any explanation of this grade-level scale. It appears that the central cities have close to average grade level in each chart so it seems that each region is individually centered. Otherwise, I'd expect to see more variability in the horizontal dots across regions.

If one scale is fixed across regions, and the other scale is adapted to each region, then we shouldn't compare the slopes across regions. The fact that the lines are generally steeper in the San Francisco chart may be an artifact of the way the scales are treated.


Finally, I'd recommend aggregating the data, and not plot individual school districts. The obsession with magnifying little details is a Big Data disease. On a chart like this, users are encouraged to click on individual districts and make inferences. However, as I discussed in the sister blog (link), most of the differences in school quality shown on these charts are not statistically meaningful (whereas the differences on the home-price scale are definitely notable). 


If you haven't already, see this related post on my sister blog for a discussion of the data analysis.





Attractive, interactive graphic challenges lazy readers

The New York Times spent a lot of effort making a nice interactive graphical feature to accompany their story about Uber's attempt to manipulate its drivers. The article is here. Below is a static screenshot of one of the graphics.


The illustrative map at the bottom is exquisite. It has Uber cars driving around, it has passengers waiting at street corners, the cars pick up passengers, new passengers appear, etc. There are also certain oddities: all the cars go at the same speed, some strange things happen when cars visually run into each other, etc.

This interactive feature is mostly concerned with entertainment. I don't think it is possible to infer either of the two metrics listed above the chart by staring at the moving Uber cars. The metrics are the percentage of Uber drivers who are idle and the average number of minutes that a passenger waits. Those two metrics are crucial to understanding the operational problem facing Uber planners. You can increase the number of Uber cars on the road to reduce average waiting time but the trade-off is a higher idle rate among drivers.


One of the key trends in interactive graphics at the Times is simplication. While a lot of things are happening behind the scenes, there is only one interactive control. The only thing the reader can control is the number of drivers in the grid.

As one of the greatest producers of interactive graphics, I trust that they know what they are doing. In fact, this article describes some comments made by Gregor Aisch, who works at the Times. The gist is: very few readers play with their interactive graphics. Someone else said, "If you make a tooltip or rollover, assume no one will ever see it." I also have heard someone say (hope this is not merely a voice in my own head): "Every extra button or knob you place on the graphic, you lose another batch of readers." This might be called the law of the interactive knob, analogous to the law of the printed equation, in the realm of popular book publishing, which stipulates that every additional equation you print in a book, you lose another batch of readers.

(Note, however, that we are talking about graphics for communications here, not exploratory graphics.)


Several years ago, I introduced the concept of "return on effort" in this blog post. Most interactive graphics are high effort to produce. The question is whether there is enough reward for the readers. 


Political winds and hair styling

Washington Post (link) and New York Times (link) published dueling charts last week, showing the swing-swang of the political winds in the U.S. Of course, you know that the pendulum has shifted riotously rightward towards Republican red in this election.

The Post focused its graphic on the urban / not urban division within the country:


Over Twitter, Lazaro Gamio told me they are calling these troll-hair charts. You certainly can see the imagery of hair blowing with the wind. In small counties (right), the wind is strongly to the right. In urban counties (left), the straight hair style has been in vogue since 2008. The numbers at the bottom of the chart drive home the story.

Previously, I discussed the Two Americas map by the NY Times, which covers a similar subject. The Times version emphasizes the geography, and is a snapshot while the Post graphic reveals longer trends.

Meanwhile, the Times published its version of a hair chart.


This particular graphic highlights the movement among the swing states. (Time moves bottom to top in this chart.) These states shifted left for Obama and marched right for Trump.

The two sets of charts have many similarities. They both use curvy lines (hair) as the main aesthetic feature. The left-right dimension is the anchor of both charts, and sways to the left or right are important tropes. In both presentations, the charts provide visual aid, and are nicely embedded within the story. Neither is intended as exploratory graphics.

But the designers diverged on many decisions, mostly in the D(ata) or V(isual) corner of the Trifecta framework.


The Times chart is at the state level while the Post uses county-level data.

The Times plots absolute values while the Post focuses on relative values (cumulative swing from the 2004 position). In the Times version, the reader can see the popular vote margin for any state in any election. The middle vertical line is keyed to the electoral vote (plurality of the popular vote in most states). It is easy to find the crossover states and times.

The Post's designer did some data transformations. Everything is indiced to 2004. Each number in the chart is the county's current leaning relative to 2004. Thus, left of vertical means said county has shifted more blue compared to 2004. The numbers are cumulative moving top to bottom. If a county is 10% left of center in the 2016 election, this effect may have come about this year, or 4 years ago, or 8 years ago, or some combination of the above. Again, left of center does not mean the county voted Democratic in that election. So, the chart must be read with some care.

One complaint about anchoring the data is the arbitrary choice of the starting year. Indeed, the Times chart goes back to 2000, another arbitrary choice. But clearly, the two teams were aiming to address slightly different variations of the key question.

There is a design advantage to anchoring the data. The Times chart is noticeably more entangled than the Post chart. There are tons more criss-crossing. This is particularly glaring given that the Times chart contains many fewer lines than the Post chart, due to state versus county.

Anchoring the data to a starting year has the effect of combing one's unruly hair. Mathematically, they are just shifting the lines so that they start at the same location, without altering the curvature. Of course, this is double-edged: the re-centering means the left-blue / right-red interpretation is co-opted.

On the Times chart, they used a different coping strategy. Each version of their charts has a filter: they highlight the set of lines to demonstrate different vignettes: the swing states moved slightly to the right, the Republican states marched right, and the Democratic states also moved right. Without these filters, the readers would be winking at the Times's bad-hair day.


Another decision worth noting: the direction of time. The Post's choice of top to bottom seems more natural to me than the Times's reverse order but I am guessing some of you may have different inclinations.

Finally, what about the thickness of the lines? The Post encoded population (voter) size while the Times used electoral votes. This decision is partly driven by the choice of state versus county level data.

One can consider electoral votes as a kind of log transformation. The effect of electorizing the popular vote is to pull the extreme values to the center. This significantly simplifies the designer's life. To wit, in the Post chart (shown nbelow), they have to apply a filter to highlight key counties, and you notice that those lines are so thick that all the other countries become barely visible.



Mapping the two Americas

If you type "two Americas map" into Google image search, you get the following top results:


Designers overwhelmingly pick the choropleth map as the way to depitct the two nations.

Now, look at these maps from the New York Times (link):


and this:


I believe the background is a relief map. Would like to see one where the color is based on the strength of support for Democrats or Republicans.

The pair of maps is extremely effective at bringing out the story about the splitting of the U.S. population. From a design standpoint, I really like it.

I love, love, love the cute annotations everywhere on the page. I imagine the designer had fun coming up with them.


Pittsburgh Puddle, Cleveland Cove, Cincinnati Slough, ...


There is an artistic (or data journalistic) license behind the way the data are processed. Most likely, a 50% cutoff is applied to determine which map a county sits atop. The analysis is at the county level so there is neccessarily some simplification... in fact, this aggregation is needed to make the "islands" and other features contiguous.

I am a bit sad that at this moment, we are so focused on what sets us apart, and not what binds us together as a nation.


PS. Via twitter, Maciej reacted negatively to these maps: "Horribly tendentious map visualization from the NYT makes the candidate who won more votes look like a tiny minority."

This is a good illustration of selecting the chart form to bring out one's message. If the goal of the chart is to show that Clinton has more votes, I agree that these maps fail to convey that message.

What I believe the NYT designer wants to point out is that the supporters of Clinton are clustered into these densely populated urban areas, leaving the Republicans with most of the land mass. (Like I said above, because of the 50% cutoff criterion, we are over-simplifying the picture. There are definitely Democrats living somewhere in Trump's nation, and likewise Republicans residing in Clinton strongholds.)

How will the Times show election results next week? Will they give us a cliffhanger?

I don't know for sure how the New York Times will present election results next week; it's going to be as hard to predict as the outcome of the election!

The Times just published a wonderful article describing all the different ways election results have been displayed in the past.

tldr; The designer has to make hard choices. Some graphics are better at one thing but worse at another. If the designer can prioritize the Qs, then the choice will come naturally. This is why the Q corner is at the top of the Trifecta framework (link).

Nytimes_election_2000I particularly like the non-map shown right, published in 2000.

This chart doesn't answer every question you want. But it gives a sense of how the candidates built their path to victory.

The imagery of a building works well here. The foundation of a building is its bottom, consisting of states which lean heavily to one party or the other. These foundational blocks scale with either the skew of the support or the number of electoral votes. The lower down in the building, the more solid is the bloc, which makes a lot of sense.

The three-tier color scheme helpfully separates partisan states, competitive states and swing states.

It's not easy to learn the exact vote totals for each state but the vertical axis is pure Tufte and sufficient for most readers.

All in all, this graphic is top-notch. It takes a little time to perfect but not too much. It has clear takeaways and I feel like I learned much more from this chart than I could in a "purple map" type of rendition.


There is a little room for augmentation. It's how they handled the "undecided" states. For me, that is the suspense of this graphic. It's the cliffhanger.

Staring at the chart for the first time, I find that it doesn't address the question of the night: who won? Neither of the "buildings" hit the 270 level required to win the election. Also, there isn't a current vote count so readers have to figure out how many votes are required to win. That's frustrating.

There is an annotation in the middle right, explaining that three states with 37 votes have not yet issued results. That text is better placed near the peaks of the buildings next to the gap where the undecided states would eventually show up.

Also, it is interesting to expand the graphic a bit to address the question of who's likely to win and how. With three states remaining that can go either way, there are eight possible scenarios. It turns out that everything comes down to Florida. Whoever wins Florida wins the election. The other two contests don't matter! (Florida has 25 votes, New Mexico 7, Oregon 5. Gore needs 16 more votes, and Bush needs 24.)

Here is one way to present these scenarios. A little bit of hover-over effect will help here, to provide some details of each scenario.





Lining up the dopers and their medals

The Times did a great job making this graphic (this snapshot is just the top half):


A lot of information is packed into a small space. It's easy to compose the story in our heads. For example, Lee Chong Wai, the Malaysian badminton silver medalist, was suspended for doping for a short time during 2015, and he was second twice before the doping incident.

They sorted the athletes according to the recency of the latest suspension. This is very smart as it helps make the chart readable. Other common ordering such as alphabetically by last name, by sport, by age, and by number of medals will result in a bit of a mess.

I'm curious about the athletes who also had doping suspensions but did not win any medals in 2016.

Brexit, Bremain, the world did not end so dataviz people can throw shade and color

Catching a dose of Alberto Cairo the other day. He has a good post about various Brexit/Bremain maps.

The story started with an editor of The Spectator, who went on twitter to make the claim that the map on the right is better than someone else's map on the left:


There are two levels at which we should discuss these maps: the scaling of the data, and the mapping of colors.

The raw data are percentages based on counts of voters so the scale is decimal. In general, we discretize the decimal data in order to improve comprehension. Discretizing means we lose granularity. This is often a good thing. The binary map on the left takes the discretization to its logical extreme. Every district is classified as either Brexit (> 50% in favor) or Bremain (> 50% opposed). The map on the right uses six total groups (so three subgroups of Brexit and three subgroups of Bremain.

Then we deal with mapping of numbers to colors. The difference between these two maps is the use of hues versus shades. The binary map uses two hues, which is probably most people's choice since we are representing two poles. The map on the right uses multiple shades of one hue. Alternatively, Alberto favors a "diverging" color scheme in which we use three shades of two hues.

The editor of The Spectator claims that his map is more "true to the data." In my view, his statement applies in these two senses: the higher granularity in the scaling, and also, the fact that there is only one data series ("share of vote for Brexit") and therefore only one color.

The second point relates to polarity of the scale. I wrote about this issue before - related to a satisfaction survey designed (not too well) by SurveyMonkey, one of the major online survey software services. In that case, I suggested that they use a bipolar instead of unipolar scale. I'd rather describe my mood as somewhat dissatisfied instead of a little bit satisfied.

I agree with Alberto here in favor of bipolarity. It's quite natural to underline the Brexit/Bremain divide.


Given what I just said, why complain about the binary map?

We agree with the editor that higher granularity improves comprehension. We just don't agree on how to add graularity. Alberto tells his readers he likes the New York Times version:


This is substantively the same map as The Spectator's, except for 8 groups instead of 6, and two hues instead of one.

Curiously enough, I gave basically the same advice to the Times regarding their maps showing U.S. Presidential primary results. I noted that their use of two hues with no shades in the Democratic race obscures the fact that none of the Democratic primiaries was a winners-take-all contest. Adding shading based on delegate votes would make the map more "truthful."

That said, I don't believe that the two improvements by the Times are sufficient. Notice that the Brexit referendum is one-person, one-vote. Thus, all of the maps above have a built-in distortion as the sizes of the regions are based on (distorted) map areas, rather than populations. For instance, the area around London is heavily Bremain but appears very small on this map.

The Guardian has a cartogram (again, courtesy of Alberto's post) which addresses this problem. Note that there is a price to pay: the shape of Great Britain is barely recognizable. But the outsized influence of London is properly acknowledged.


 This one has two hues and four shades.  For me, it is most "truthful" because the sizes of the colored regions are properly mapped to the vote proportions.

A multidimensional graphic that holds a number of surprises, via NYT

The New York Times has an eye-catching graphic illustrating the Amtrak crash last year near Philadelphia. The article is here.

The various images associated with this article vary in the amount of contextual details offered to readers.

This graphic provides an overview of the situation:


Initially, I had a fair amount of trouble deciphering this chart. I was searching hard to find the contrast between the orange (labeled RECENT TRAINS) and the red (labeled TRAIN # 188). The orange color forms a wavy area akin to a river on a map. The red line segments suggest bridges that span the river bank. The visual cues kept telling me train #188 is a typical train but that conclusion was obviously wrong.

The confusion went away after I read the next graphic:


This zoomed-in view offered some helpful annotation. The data came from three days of trains prior to the accident. Surprisingly, the orange band does not visualize a range of speeds. The width of the orange band fluctuates with the median speed over those three days. And then, the red line segments represent the speed of train #188 as it passed through specific points on the itinerary.

The key visual element to look for is the red lines exceeding the width of the orange band as train #188 rounds Frankford Junction.


In the second graphic, the speeding is more visible. But it can be made even more prominent. For example, instead of line segments, use the same curvy element to portray the speed of train #188. Then through line width or color, emphasize train #188 and push the average train to the background.


Notice that there is an additional line snaking through the middle of the orange band. The data have been centered around this line. This type of centering is problematic: the excess speed relative to the median train has been split into halves. The reader must mentally reassemble the halves. The impact of the speeding has therefore been artificially muted.

 In this next version, I keep that midpoint line and use it to indicate the median speed of the trains. Then, I show how train #188 diverged from the median speed as it neared the Junction.



 This version brings out one other confusing element of the original. This line that traces the median speed is also tracing the path of the train (geographically). Actually, the line does not encode speed--it just encodes the reference level of speed. The graphic above creates an impression that train #188 "ran off track" if the reader interprets the green line as a railroad track on a map. But it is off in speed, not in physical location.




Delegate maps need a color treatment

This year's U.S. primary elections have been very entertaining. Delegate maps are a handy way to keep track of the horse race. They provide data to support (or refute) the narratives created by reporters who use words like "landslide", "commanding", etc.

Here’s a delegate map used by the New York Times on the night of Mar 15th when Hillary Clinton won four out of five states, with the fifth (Missouri) being a cliffhanger:


Other media outlets are using pretty much the same form, with different color schemes.

The typical color scheme has two binary levels: one color for each candidate (NYT uses blue for Clinton, green for Sanders in the Democratic race); a lighter shade for who's leading, and a darker shade for the declared winner.


These maps are missing one crucial piece of information, the margin of victory. The margin is important because in most of the contests, the delegates are split proportionally.

The same shade of blue was used to describe the decisive victory in Florida (64% to 33%) and the laser-thin victory in Illinois (51% to 49%). This color scheme implies a winner-takes-all criterion.


Here is a map that includes the margin of victory, computed as the excess number of delegates won in the given state:


For the Democratic race, the narrative is that Clinton built a sizeable lead in pledged delegates in the Southern states; elsewhere, the states have been evenly split or slightly favoring Sanders (within about 10 delegates). Also, the West and Northwest have largely not spoken yet.

Other maps can be created using different measures of the margin, such as the diference in vote proportions.

I prefer color schemes that reveal the delegate allocation criterion.