A report came out from Stanford School of Medicine about a study of Apple Watch's health monitoring features. Some headline writers are proclaiming that "finally, there is proof that these watches benefit our health!" For example, Apple Watch Stanford Study Shows How It Can Save Lives (link).
When you read the official story, you will learn the following facts about the study:
- The research is funded by Apple
- It was a purely observational study in which they follow (400,000) people who wear Apple Watches
- Participants must own both an Apple Watch and an iPhone to be eligible (plus meeting other criteria)
- There was no "control" group - they did not follow anyone who did not use Apple Watch or use any other health monitoring wearables
- Every participant is self-selected
- The device issued warnings to only 0.5 percent of the participants (~ 2,160)
- Those who received a warning were directed to a video consultation; and the doctor decided whether or not to send the participant an ECG patch, which is used to establish the "ground truth". Only about 30 percent were sent patches, and of those, 70 percent (450) returned the patches for analysis.
- Only those who had ECG data were analyzed. One third of these were shown to have experienced "atrial fibrillation" (irregular heartbeat). This means that two-thirds got false alarms. But if we include the 70% who were not sent patches after the video consultation as false alarms as well, then out of every 100 warnings, only 7 were validated.
- There is no discussion of false negatives: did any of the 99.5 percent who did not receive warnings experienced irregular heartbeats?
- We do know that if there were significant false negatives, then more warnings would have to be sent, which pushes up false alarms.
- Despite the headlines, any lives saved were extrapolated.
There are some major methodological limitations about this study.
Firstly, the study design prevents drawing conclusions of the type "People wearing Apple Watches .... compared to those who did not wear Apple Watches." It does not include anyone not wearing Apple Watches.
Secondly, it's difficult to interpret the accuracy metrics. Is 20 percent false alarms a good or bad number? Is 0.5 percent receiving warnings a reasonable proportion given the demographic and health characteristics of the study population?
Hopefully, this study is just the beginning, and more rigorous studies are being planned.
Comments