validity/reliability of consumer-wearable activity trackers
Posted: Tue Dec 22, 2015 2:09 pm
There has been several threads about fitness trackers and their reliability. Here is the latest review, which I think does a good job. The full text is available at the link.
From my own experience, and I am both a numbers and gadget guy so I love to play with these things, here is what I have found.
There is a wide variance in accuracy and reliability. I can wear 5 pedometer and walk 3 miles and they will all register different (and yes, I have done that). However, the better ones (as mentioned in the article) are fairly accurate in regard to steps and for the ones that can also calculate it, stairs to. Distance can be accurate if you are willing to accurately calculate your stride length and input it. However, depending on you walking or running speed, stride length changes, so you have to put in a length based on your average walking and/or running pace. Some allow you to enter a value for both.
If the tracker has GPS, then it is much more accurate in relation to distance.
In regard to HR, I have found nothing as accurate as the chest strap versions of HR monitors and find most of the main brands (Garmin, Polar, etc) very accurate. The trackers that do monitor HR, including the Apple Watch, use a pulsating light to detect your capillaries expanding and contracting based on blood volume changes with each heart beat. I have found this to have fairly good accuracy when one is not doing much activity. However, the more active and the more intense the activity and the more variance in the activity (Intervals, sprints, etc), I found them to be fair to poor. I have worn several while also wearing a chest strap monitor and they just do not do as good a job and the difference can be huge.
What I did like about the HR on the activity tracker is they can measure it 24 hours a day so it was good to see what it was during true rest, i.e., like while resting and/or sleeping. It clocked mine at 44-50 bpm during the middle of the night consistently over time, which is about the same when I have checked it against a Chest Strap monitor.
Considering all the above, there is no way they are very accurate in regard to calories as there is too much variance and variables involved.
Currently, I own the Polar HR7 for heart rate monitoring during exercise.
For activity & steps (and fun!), my wife and I both have the Fitbit One, which I have calibrated using both simultaneous GPS and actual mileage and after doing that, find it to have really good accuracy. I tried several of the other trackers including several other FitBit models, and because of the inaccuracy in the HR and the fact I don't like to wear something on my wrist all day, I find the FitBit one to be the best choice. Studies have confirmed its accuracy.
Before pedometers went high tech, my favorite was the Omron 112, which was tested and shown to be very accurate but may no longer be available.
The above is just my opinion based on my experience over the years.
The study below reviews the current data/science on them.
In Health
Jeff
Review
Systematic review of the validity and reliability of consumer-wearable activity trackers
International Journal of Behavioral Nutrition and Physical Activity 2015, 12:159
doi:10.1186/s12966-015-0314-1
http://www.ijbnpa.org/content/12/1/159
Abstract
Background
Consumer-wearable activity trackers are electronic devices used for monitoring fitness- and other health-related metrics. The purpose of this systematic review was to summarize the evidence for validity and reliability of popular consumer-wearable activity trackers (Fitbit and Jawbone) and their ability to estimate steps, distance, physical activity, energy expenditure, and sleep.
Methods
Searches included only full-length English language studies published in PubMed, Embase, SPORTDiscus, and Google Scholar through July 31, 2015. Two people reviewed and abstracted each included study.
Results
In total, 22 studies were included in the review (20 on adults, 2 on youth). For laboratory-based studies using step counting or accelerometer steps, the correlation with tracker-assessed steps was high for both Fitbit and Jawbone (Pearson or intraclass correlation coefficients (CC) > =0.80). Only one study assessed distance for the Fitbit, finding an over-estimate at slower speeds and under-estimate at faster speeds. Two field-based studies compared accelerometry-assessed physical activity to the trackers, with one study finding higher correlation (Spearman CC 0.86, Fitbit) while another study found a wide range in correlation (intraclass CC 0.36–0.70, Fitbit and Jawbone). Using several different comparison measures (indirect and direct calorimetry, accelerometry, self-report), energy expenditure was more often under-estimated by either tracker. Total sleep time and sleep efficiency were over-estimated and wake after sleep onset was under-estimated comparing metrics from polysomnography to either tracker using a normal mode setting. No studies of intradevice reliability were found. Interdevice reliability was reported on seven studies using the Fitbit, but none for the Jawbone. Walking- and running-based Fitbit trials indicated consistently high interdevice reliability for steps (Pearson and intraclass CC 0.76–1.00), distance (intraclass CC 0.90–0.99), and energy expenditure (Pearson and intraclass CC 0.71–0.97). When wearing two Fitbits while sleeping, consistency between the devices was high.
Conclusion
This systematic review indicated higher validity of steps, few studies on distance and physical activity, and lower validity for energy expenditure and sleep. The evidence reviewed indicated high interdevice reliability for steps, distance, energy expenditure, and sleep for certain Fitbit models. As new activity trackers and features are introduced to the market, documentation of the measurement properties can guide their use in research settings.
From my own experience, and I am both a numbers and gadget guy so I love to play with these things, here is what I have found.
There is a wide variance in accuracy and reliability. I can wear 5 pedometer and walk 3 miles and they will all register different (and yes, I have done that). However, the better ones (as mentioned in the article) are fairly accurate in regard to steps and for the ones that can also calculate it, stairs to. Distance can be accurate if you are willing to accurately calculate your stride length and input it. However, depending on you walking or running speed, stride length changes, so you have to put in a length based on your average walking and/or running pace. Some allow you to enter a value for both.
If the tracker has GPS, then it is much more accurate in relation to distance.
In regard to HR, I have found nothing as accurate as the chest strap versions of HR monitors and find most of the main brands (Garmin, Polar, etc) very accurate. The trackers that do monitor HR, including the Apple Watch, use a pulsating light to detect your capillaries expanding and contracting based on blood volume changes with each heart beat. I have found this to have fairly good accuracy when one is not doing much activity. However, the more active and the more intense the activity and the more variance in the activity (Intervals, sprints, etc), I found them to be fair to poor. I have worn several while also wearing a chest strap monitor and they just do not do as good a job and the difference can be huge.
What I did like about the HR on the activity tracker is they can measure it 24 hours a day so it was good to see what it was during true rest, i.e., like while resting and/or sleeping. It clocked mine at 44-50 bpm during the middle of the night consistently over time, which is about the same when I have checked it against a Chest Strap monitor.
Considering all the above, there is no way they are very accurate in regard to calories as there is too much variance and variables involved.
Currently, I own the Polar HR7 for heart rate monitoring during exercise.
For activity & steps (and fun!), my wife and I both have the Fitbit One, which I have calibrated using both simultaneous GPS and actual mileage and after doing that, find it to have really good accuracy. I tried several of the other trackers including several other FitBit models, and because of the inaccuracy in the HR and the fact I don't like to wear something on my wrist all day, I find the FitBit one to be the best choice. Studies have confirmed its accuracy.
Before pedometers went high tech, my favorite was the Omron 112, which was tested and shown to be very accurate but may no longer be available.
The above is just my opinion based on my experience over the years.
The study below reviews the current data/science on them.
In Health
Jeff
Review
Systematic review of the validity and reliability of consumer-wearable activity trackers
International Journal of Behavioral Nutrition and Physical Activity 2015, 12:159
doi:10.1186/s12966-015-0314-1
http://www.ijbnpa.org/content/12/1/159
Abstract
Background
Consumer-wearable activity trackers are electronic devices used for monitoring fitness- and other health-related metrics. The purpose of this systematic review was to summarize the evidence for validity and reliability of popular consumer-wearable activity trackers (Fitbit and Jawbone) and their ability to estimate steps, distance, physical activity, energy expenditure, and sleep.
Methods
Searches included only full-length English language studies published in PubMed, Embase, SPORTDiscus, and Google Scholar through July 31, 2015. Two people reviewed and abstracted each included study.
Results
In total, 22 studies were included in the review (20 on adults, 2 on youth). For laboratory-based studies using step counting or accelerometer steps, the correlation with tracker-assessed steps was high for both Fitbit and Jawbone (Pearson or intraclass correlation coefficients (CC) > =0.80). Only one study assessed distance for the Fitbit, finding an over-estimate at slower speeds and under-estimate at faster speeds. Two field-based studies compared accelerometry-assessed physical activity to the trackers, with one study finding higher correlation (Spearman CC 0.86, Fitbit) while another study found a wide range in correlation (intraclass CC 0.36–0.70, Fitbit and Jawbone). Using several different comparison measures (indirect and direct calorimetry, accelerometry, self-report), energy expenditure was more often under-estimated by either tracker. Total sleep time and sleep efficiency were over-estimated and wake after sleep onset was under-estimated comparing metrics from polysomnography to either tracker using a normal mode setting. No studies of intradevice reliability were found. Interdevice reliability was reported on seven studies using the Fitbit, but none for the Jawbone. Walking- and running-based Fitbit trials indicated consistently high interdevice reliability for steps (Pearson and intraclass CC 0.76–1.00), distance (intraclass CC 0.90–0.99), and energy expenditure (Pearson and intraclass CC 0.71–0.97). When wearing two Fitbits while sleeping, consistency between the devices was high.
Conclusion
This systematic review indicated higher validity of steps, few studies on distance and physical activity, and lower validity for energy expenditure and sleep. The evidence reviewed indicated high interdevice reliability for steps, distance, energy expenditure, and sleep for certain Fitbit models. As new activity trackers and features are introduced to the market, documentation of the measurement properties can guide their use in research settings.