While running down the rabbit hole following a recent post at the Gelman blog, I found an appetizing quote from Martha S. on a post from 2016 (link).
The topic is ostensibly about the Big Five personality test. One of the five dimensions is named "openness to experience". Martha wrote:
If I’m not mistaken, the label “openness to experience” was an ex-post-facto label that then became taken as a definition. To elaborate, my understanding is that the process was as follows:
The initial research started not by making “definitions” of personality traits, but by making lists of questions that the researchers thought might capture the varied aspects of personality. They administered the resulting questionnaire to a group of people and ran a factor analysis on the resulting data. This spit out a list of “factors” (linear combinations of the question responses) that “accounted for” (relatively) high proportions of the total variance in responses. They decided that the first five (ie, the five with the highest amount of total variance accounted for) gave a pretty good list of the major “traits” of personality. They then gave names to these traits.
In other words, they did not start with definitions of traits; this was exploratory research that gave them candidates for traits. The real definition of the traits was “whatever this linear combination measures”. However, the labels they attached to these factors became “reified” — that is, taken to be The Real Thing Measured, even thought the labels were fuzzy terms subject to varying interpretations.
***
I think this quote is very relevant for anyone who's working in the area of "interpretable" models, which has become particularly popular in the deep learning community. Martha's point is that the Big Five personality traits is "an" interpretation of the model (linear combinations), which then got reified into "the" interpretation.
Martha was responding to a comment by another reader Diana S. who complained that the Big Five dimension "openness to experience" is not well defined. She wrote:
researchers seem to take this to mean “openness to an ever-changing array of experiences.” That is, someone who reads five different books would be considered more “open to experience” than someone who reads the same book five times–even if the latter person finds new things with each rereading. As I see it, someone who rereads may be at least as open to experience as the person always reaching for a new book–but the quality manifests itself differently.
This comment was made to illuminate why we need to have precise definitions of things we're measuring.
The generally accepted definition Diana cited has strayed from how the original dimension was conceived. In fact, Martha suggested that the original definition does not actually exist. Well, it exists in a purely mathematical - abstract - form: the precise definition in the original analysis would be the linear combination of responses to the test, i.e. a set of coefficients. But this precise definition is devoid of practical meaning. How does one interpret 5.3258*Q1 vs 6.2335*Q1?
***
Also relevant is Efron's paper which did a number on measures of "importance" that get pumped out with every machine learning black box model. See my previous post here.
Comments