Superb tile map offering multiple avenues for exploration

Here's a beauty by WSJ Graphics:

The article is here.

This data graphic illustrates the power of the visual medium. The underlying dataset is complex: power production by type of source by state by month by year. That's more than 90,000 numbers. They all reside on this graphic.

Readers amazingly make sense of all these numbers without much effort.

It starts with the summary chart on top.

The designer made decisions. The data are presented in relative terms, as proportion of total power production. Only the first and last years are labeled, thus drawing our attention to the long-term trend. The order of the color blocks is carefully selected so that the cleaner sources are listed at the top and the dirtier sources at the bottom. The order of the legend labels mirrors the color blocks in the area chart.

It takes only a few seconds to learn that U.S. power production has largely shifted away from coal with most of it substituted by natural gas. Other than wind, the green sources of power have not gained much ground during these years - in a relative sense.

This summary chart serves as a reading guide for the rest of the chart, which is a tile map of all fifty states. Embedded in the tile map is a small-multiples arrangement.

***

The map offers multiple avenues for exploration.

Some readers may look at specific states. For example, California.

Currently, about half of the power production in California come from natural gas. Notably, there is no coal at all in any of these years. In addition to wind, solar energy has also gained. All of these insights come without the need for any labels or gridlines!

Browsing around California, readers find different patterns in other Western states like Oregon and Washington.

Hydroelectric energy is the dominant source in those two states, with wind gradually taking share.

At this point, readers realize that the summary chart up top hides remarkable state-level variations.

***

There are other paths through the map.

Some readers may scan the whole map, seeking patterns that pop out.

One such pattern is the cluster of states that use coal. In most of these states, the proportion of coal has declined.

Yet another path exists for those interested in specific sources of power.

For example, the trend in nuclear power usage is easily followed by tracking the purple. South Carolina, Illinois and New Hampshire are three states that rely on nuclear for more than half of its power.

I wonder what happened in Vermont about 8 years ago.

The chart says they renounced nuclear energy. Here is some history. This one-time event caused a disruption in the time series, unique on the entire map.

***

This work is wonderful. Enjoy it!

There's more to the composite rating chart

In my previous post, I sketched a set of charts to illustrate composite ratings of maps platforms (e.g. Google Maps, TomTom). Here is the sketch again:

For those readers who are interested in understanding these ratings beyond the obvious, this set of charts has more to offer.

Take a look first at the two charts on the left hand side.

Compare the patterns of dots between the two charts. You should note that the Maps Data ratings (blue dots) are less variable than the Platform ratings (green dots).

For Maps Data, the range is from 30 to 85 (out of 110) but the majority of the dots line up around 50.

For Platform, the range is 20 to 70 (out of 90) and the dots are quite spread out within this range.

This means competitiveness based on Platform is more differentiating among these brands than is Maps Data.

In the previous post, I already noted that the other key insight is that the Maps Data values hang quite closely to the overall average ratings while the Platform values are much less correlated.

***

Another informative observation can be found in the bottom row of charts.

The yellow dots (Developer Ecosystem) are mostly to the right of the overall ratings, meaning most of these brands were given scores on Developer Ecosystem that are higher than their average scores.

That is not the case with the green dots (Platform). For this sub-rating, most of the brands score lower than they do in the overall rating.

***

None of these insights are readily learned from the stacked column chart. A key skill in data visualization is whether one can pile on insights without overloading the chart.

 

 

Visualizing composite ratings

A twitter reader submitted the following chart from Autoevolution (link):

This is not a successful chart for the simple reason that readers want to look away from it. It's too busy. There is so much going on that one doesn't know where to look.

The underlying dataset is quite common in the marketing world. Through surveys, people are asked to rate some product along a number of dimensions (here, seven). Each dimension has a weight, and combined, the weighted sum becomes a composite ranking (shown here in gray).

Nothing in the chart stands out as particularly offensive even though the overall effect is repelling. Adding the overall rating on top of each column is not the best idea as it distorts the perception of the column heights. But with all these ingredients, the food comes out bland.

***

The key is editing. Find the stories you want to tell, and then deconstruct the chart to showcase them.

I start with a simple way to show the composite ranking, without any fuss:

[Since these are mockups, I have copied all of the data, just the top 11 items.]

Then, I want to know if individual products have particular strengths or weaknesses along specific dimensions. In a ranking like this, one should expect that some component ratings correlate highly with the overall rating while other components deviate from the overall average.

An example of correlated ratings is the Customers dimension.

The general pattern of the red dots clings closely to that of the gray bars. The gray bars are the overall composite ratings (re-scaled to the rating range for the Customers dimension). This dimension does not tell us more than what we know from the composite rating.

By contrast, the Developers Ecosystem dimension provides additional information.

Esri, AzureMaps and Mapbox performed much better on this dimension than on the average dimension. 

***

The following construction puts everything together in one package:

Illustrating coronavirus waves with moving images

The New York Times put out a master class in visualizing space and time data recently, in a visualization of five waves of Covid-19 that have torched the U.S. thus far (link).

The project displays one dataset using three designs, which provides an opportunity to compare and contrast them.

***

The first design - above the headline - is an animated choropleth map. This is a straightforward presentation of space and time data. The level of cases in each county is indicated by color, dividing the country into 12 levels (plus unknown). Time is run forward. The time legend plays double duty as a line chart that shows the change in the weekly rate of reported cases over the course of the pandemic. A small piece of interactivity binds the legend with the map.

(To see a screen recording of the animation, click on the image above.)

***

The second design comprises six panels, snapshots that capture crucial "turning points" during the Covid-19 pandemic. The color of each county now encodes an average case rate (I hope they didn't just average the daily rates). 

The line-chart legend is gone -  it's not hard to see Winter > Fall 2020 > Summer/Fall 2021 >... so I don't think it's a big loss.

The small-multiples setup is particularly effective at facilitating comparisons: across time, and across space. It presents a story in pictures.

They may have left off 2020 following "Winter" because December to February spans both years but "Winter 2020" may do more benefit than harm here.

***

The third design is a series of short films, which stands mid-way between the single animated map and the six snapshots. Each movie covers a separate window of time.

This design does a better job telling the story within each time window while it obstructs comparisons across time windows.

The informative legend is back. This time, it's showing the static time window for each map.

***

The three designs come from the same dataset. I think of them as one long movie, six snapshots, and five short films.

The one long movie is a like a data dump. It shows every number in the dataset, which is the weekly case rate for each county for a given week. All the data are streamed into a single map. It's a show piece.

As an instrument to help readers understand the patterns in the dataset, the movie falls short. Too much is going on, making it hard to focus and pick out key trends. When your eyes are everywhere, they are nowhere.

The six snapshots represent the other extreme. The graph does not move, as the time axis is reduced to six discrete time points. But this display describes the change points, and tells a story. The long movie, by contrast, invites readers to find a story.

Without motion, the small-multiples format allows us to pick out specific counties or regions and compare the case rates across time. This task is close to impossible in the long movie, as it requires freezing the movie, and jumping back and forth.

The five short films may be the best of both worlds. It retains the motion. If the time windows are chosen wisely, each short film contains a few simple patterns that can easily be discerned. For example, the third film shows how the winter wave emerged from the midwest and then walloped the whole country, spreading southward and toward the coasts.

(If the above gif doesn't play, click it.)

***

If there is double or triple the time allocated to this project, I'd want to explore spatial clustering. I'd like to dampen the spatial noise (neighboring counties that have slightly different experiences). There is also temporal noise (fluctuations from week to week for the same county) - which can be smoothed away. I think with these statistical techniques, the "wave" feature of the pandemic may be more visible.

 

 

The gift of small edits and subtraction

While making the chart on fertility rates (link), I came across a problem that pops up quite often, and is  ignored by most software programs.

Here is an earlier version of the chart I later discarded:

Compare this to the version I published in the blog post:

Aside from adding the chart title, there is one major change. I removed the empty plots from the grid. This is a visualization trick that should be called adding by subtracting. The empty scaffolding on the first chart increases our cognitive load without yielding any benefit. The whitespace brings out the message that only countries in Asia and Africa have fertility rates above 5.0. 

This is a small edit. But small edits accumulate and deliver a big impact. Bear this in mind the next time you make a chart.

 

P.S.

(1) You'd have to use a lower-level coding language to execute this small edit. Most software programs are quite rigid when it comes to making small-multiples (facet) charts.

(2) If there is a next iteration, I'd reverse the Asia and Oceania rows.

 

Visualizing fertility rates around the globe

The following chart dropped on my Twitter feed.

It's an ambitious chart that tries to do a lot. The underlying data set contains fertility rate data from over 200 countries over 20 years.

The basic chart form is a column chart that is curled up into a ball. The column chart is given colors that map to continents. All countries are grouped into five continents. The column chart can only take a single data series, so the 2019 fertility rate is chosen.

Beyond this basic setup, the designer embellishes the chart with a trove of information. Here's a close up:

The first number is the 2019 fertility rate, which means all the data encoded into the columns are also printed on the chart itself. Then, the flag of each country forms the next ring. Then, the name of the country. Finally, in brackets, the percent change in fertility rate between 2000 and 2019.

That is not all. Some contextual information are injected in those arrows that connect the columns to the data labels. A green arrow indicates that the fertility rate is trending lower - which is the case in most countries around the world. Once in a while, a purple arrow pops up. In the above excerpt, Seychelles gets a purple arrow because this island nation has increased the fertility rate from 2000 to 2019.

Also hiding in the background are several dashed rings. I think only the one that partially overlaps with the column chart contains any information - the other rings are inserted for an artistic reason. To decipher this dashed ring, we must look at the inset in the top left corner. We learn that the value of 2.1 children per woman is known as the replacement fertility rate. So it's also possible to assess whether each country is above or below the replacement fertility rate threshold.

[I'm presuming that this replacement threshold is about the births necessary to avoid a population decline. If that's the case, then comparing each country's fertility rate to a global fertility rate threshold is too simplistic because fertility is only one of several key factors driving a country's population growth. A more sophisticated model should generate country-level thresholds.]

***

Data graphics serve many functions. This chart works well as an embellished data table. It does take some time to find a specific country because the columns have been sorted by decreasing 2019 fertility rate but once we locate the column, all the other data fields are clearly laid out.

As a generator of data insights, this chart is less effective. The main insight I obtained from it is a rough ranking of continents, with African countries predominantly having higher fertility rates, followed by Asia and Oceania, then Americas, and finally, Europe which has the lowest fertility rates. If this is the key message, a standard choropleth map brings it out more directly.

***

Here is a small-multiples rendering of the fertility dataset. I chose 1999 values instead of 2000 to make a complete two-decade view.

The columns represent a grouping of countries based on their 1999 fertility rates. The left column contains countries with the lowest number of births per woman, and the fertility rate increases left to right - both within an individual plot and in the grid.

If you're wondering, the hidden vertical axis sorts the countries by their 1999 rank. The lighter colors are 1999 values while the darker colors are 2019 values. For most countries the dots are shifting left over the 20 years. There are some exceptions. I have labeled several of these exceptions (e.g. Kazakhstan and Mongolia), and rendered them in italic.

 

 

 

Ridings, polls, elections, O Canada

Stephen Taylor reached out to me about his work to visualize Canadian elections data. I took a look. I appreciate the labor of love behind this project.

He led with a streamgraph, which presents a quick overview of relative party strengths over time.

I am no Canadian election expert, and I did a bare minimum of research in writing this blog. From this chart, I learn that:

• the Canadians have an irregular election schedule
• Canada has a two party plus breadcrumbs system
• The two dominant parties are Liberals and Conservatives. The Liberals currently hold just less than half of the seats. The Conservatives have more than half of the seats not held by Liberals
• The Conservative party (maybe) rebranded as "progressive conservative" for several decades. The Reform/Alliance party was (maybe) a splinter movement within the Conservatives as well.
• Since the "width" of the entire stream increased over time, I'm guessing the number of seats has expanded

That's quite a bit of information obtained at a glance. This shows the power of data visualization. Notice Stephen didn't even have to include a "how to read this" box.

The streamgraph form has its limitations.

The feature that makes it more attractive than an area chart is its middle anchoring, resulting in a form of symmetry. The same feature produces erroneous intuition - the red patch draws out a declining trend; the reader must fight the urge to interpret the lines and focus on the areas.

The breadcrumbs are well hidden. The legend below discloses that the Green Party holds 3 seats currently. The party has never held enough seats to appear on the streamgraph though.

The bars showing proportions in the legend is a very nice touch. (The numbers appear messed up - I have to ask Stephen whether the seats shown are current values, or some kind of historical average.) I am a big fan of informative legends.

***

The next featured chart is a dot plot of polling results since 2020.

One can see a three-tier system: the two main parties, then the NDP (yellow) is the clear majority of the minority, and finally you have a host of parties that don't poll over 10%.

It looks like the polls are favoring the Conservatives over the Liberals in this election but it may be an election-day toss-up.

The purple dots represent "PPC" which is a party not found elsewhere on the page.

This chart is clear as crystal because of the structure of the underlying data. It just amazes me that the polls are so highly correlated. For example, across all these polls, the NDP has never once polled better than either the Liberals or the Conservatives, and in addition, it has never polled worse than any of the small parties.

What I'd like to see is a chart that merges the two datasets, addressing the question of how well these polls predicted the actual election outcomes.

***

The project goes very deep as Stephen provides charts for individual "ridings" (perhaps similar to U.S. precincts).

Here we see population pyramids for Vancouver Center, versus British Columbia (Province), versus Canada.

This riding has a large surplus of younger people in their twenties and thirties. Be careful about the changing scales though. The relative difference in proportions are more drastic than visually displayed because the maximum values (5%) on the Province and Canada charts are half that on the Riding chart (10%). Imagine squashing the Province and Canada charts to half their widths.

Analyses of income and rent/own status are also provided.

This part of the dashboard exhibits a problem common in most dashboards - they present each dimension of the data separately and miss out on the more interesting stuff: the correlation between dimensions. Do people in their twenties and thirties favor specific parties? Do richer people vote for certain parties?

***

The riding-level maps are the least polished part of the site. This is where I'm looking for a "how to read it" box.

It took me a while to realize that the colors represent the parties. If I haven't come in from the front page, I'd have been totally lost.

Next, I got confused by the use of the word "poll". Clicking on any of the subdivisions bring up details of an actual race, with party colors, candidates and a donut chart showing proportions. The title gives a "poll id" and the name of the riding in parentheses. Since the poll id changes as I mouse over different subdivisions, I'm wondering whether a "poll" is the term for a subdivision of a riding. A quick wiki search indicates otherwise.

My best guess is the subdivisions are indicated by the numbers.

Back to the donut charts, I prefer a different sorting of the candidates. For this chart, the two most logical orderings are (a) order by overall popularity of the parties, fixed for all ridings and (b) order by popularity of the candidate, variable for each riding.

The map shown above gives the winner in each subdivision. This type of visualization dumps a lot of information. Stephen tackles this issue by offering a small multiples view of each party. Here is the Liberals in Vancouver.

Again, we encounter ambiguity about the color scheme. Liberals have been associated with a red color but we are faced with abundant yellow. After clicking on the other parties, you get the idea that he has switched to a divergent continuous color scale (red - yellow - green). Is red or green the higher value? (The answer is red.)

I'd suggest using a gray scale for these charts. The hardest decision is going to be the encoding between values and shading. Should each gray scale be different for each riding and each party?

If I were to take a guess, Stephen must have spent weeks if not months creating these maps (depending on whether he's full-time or part-time). What he has published here is a great start. Fine-tuning the issues I've mentioned may take more weeks or months more.

****

Stephen is brave and smart to send this project for review. For one thing, he's got some free consulting. More importantly, we should always send work around for feedback; other readers can tell us where our blind spots are.

To read more, start with this post by Stephen in which he introduces his project.

Tongue in cheek but a master stroke

Andrew jumped on the Benford bandwagon to do a tongue-in-cheek analysis of numbers in Hollywood movies (link). The key graphic is this:

Benford's Law is frequently invoked to prove (or disprove) fraud with numbers by examining the distribution of first digits. Andrew extracted movies that contain numbers in their names - mostly but not always sequences of movies with sequels. The above histogram (gray columns) are the number of movies with specific first digits. The red line is the expected number if Benford's Law holds. As typical of such analysis, the histogram is closely aligned with the red line, and therefore, he did not find any fraud. 

I'll blog about my reservations about Benford-style analysis on the book blog later - one quick point is: as with any statistical analysis, we should say there is no statistical evidence of fraud (more precisely, of the kind of fraud that can be discovered using Benford's Law), which is different from saying there is no fraud.

***

Andrew also showed a small-multiples chart that breaks up the above chart by movie groups. I excerpted the top left section of the chart below:

The genius in this graphic is easily missed.

Notice that the red lines (which are the expected values if Benford Law holds) appear identical on every single plot. And then notice that the lines don't represent the same values.

It's great to have the red lines look the same everywhere because they represent the immutable Benford reference. Because the number of movies is so small, he's plotting counts instead of proportions. If you let the software decide on the best y-axis range for each plot, the red lines will look different on different charts!

You can find the trick in the R code from Gelman's blog.

First, the maximum value of each plot is set to the total number of observations. Then, the expected Benford proportions are converted into expected Benford counts. The first Benford count is then shown against an axis topping out at the total count, and thus, relatively, what we are seeing are the Benford proportions. Thus, every red line looks the same despite holding different values.

This is a master stroke.

 

 

 

Did prices go up or down? Depends on how one looks at the data

The U.S. media have been flooded with reports of runaway inflation recently, and it's refreshing to see a nice article in the Wall Street Journal that takes a second look at the data. Because as my readers know, raw data can be incredibly deceptive.

Inflation typically describes the change in price level relative to the prior year. The month-on-month change in price levels is a simple seasonal adjustment used to remove the effect of seasonality that masks the true change in price levels. (See this explainer of seasonal adjustment.)

As the pandemic enters the second year, this methodology is comparing 2021 price levels to pandemic-impacted price levels of 2020. This produces a very confusing picture. As the WSJ article explains, prices can be lower than they were in 2019 (pre-pandemic) and yet substantially higher than they were in 2020 (during the pandemic). This happens in industry sectors that were heavily affected by the economic shutdown, e.g. hotels, travel, entertainment.

Here is how they visualized this phenomenon. Amusingly, some algorithm estimated that it should take 5 minutes to read the entire article. It may take that much time to understand properly what this chart is showing.

Let me save you some time.

The chart shows monthly inflation rates of hotel price levels.

The pink horizontal stripes represent the official inflation numbers, which compare each month's hotel prices to those of a year prior. The most recent value for May of 2021 says hotel prices rose by 9% compared to May of 2020.

The blue horizontal stripes show an alternative calculation which compares each month's hotel prices to those of two years prior. Think of 2018-9 as "normal" years, pre-pandemic. Using this measure, we find that hotel prices for May of 2021 are about 4% lower than for May of 2019.

(This situation affects all of our economic statistics. We may see an expansion in employment levels from a year ago which still leaves us behind where we were before the pandemic.)

What confused me on the WSJ chart are the blocks of color. In a previous chart, the readers learn that solid colors mean inflation rose while diagonal lines mean inflation decreased. It turns out that these are month-over-month changes in inflation rates (notice that one end of the column for the previous month touches one end of the column of the next month).

The color patterns become the most dominant feature of this chart, and yet the month-over-month change in inflation rates isn't the crux of the story. The real star of the story should be the difference in inflation rates - for any given month - between two reference years.

***

In the following chart, I focus attention on the within-month, between-reference-years comparisons.

Because hotel prices dropped drastically during the pandemic, and have recovered quite well in recent months as the U.S. reopens the economy, the inflation rate of hotel prices is almost 10%. Nevertheless, the current price level is still 7% below the pre-pandemic level.

 

 

Probabilities and proportions: which one is the chart showing

The New York Times showed this chart (link):

My first read: oh my gosh, 40-50% of the unvaccinated Americans are living their normal lives - dining at restaurants, assembling with more than 10 people, going to religious gatherings.

After reading the text around this chart, I realize I have misinterpreted it.

The chart should be read by columns. Each column is a "pie chart". For example, the first column shows that half the restaurant diners are not vaccinated, a third are fully vaccinated, and the remainder are partially vaccinated. The other columns have roughly the same proportions.

The author says "The rates of vaccination among people doing these activities largely reflect the rates in the population." This line is perhaps more confusing than intended. What she's saying is that in the general population, half of us are unvaccinated, a third are fully unvaccinated, and the remainder are partially vaccinated.

Here's a picture:

What this chart is saying is that the people dining out is like a random sample from all Americans. So too the other groups depicted. What Americans are choosing to do is independent of their vaccination status.

Unvaccinated people are no less likely to be doing all these activities than the fully vaccinated. This raises the question: are half of the people not wearing masks outdoors unvaccinated?

***

Why did I read the chart wrongly in the first place? It has to do with expectations.

Most survey charts plot probabilities not proportions. I haphazardly grabbed the following Pew Research chart as an example:

From this chart, we learn that 30% of kids 9-11 years old uses TikTok compared to 11% of kids 5-8.  The percentages down a column do not sum to 100%.