validity/reliability of consumer-wearable activity trackers

[JeffN]

There has been several threads about fitness trackers and their reliability. Here is the latest review, which I think does a good job. The full text is available at the link.

From my own experience, and I am both a numbers and gadget guy so I love to play with these things, here is what I have found.

There is a wide variance in accuracy and reliability. I can wear 5 pedometer and walk 3 miles and they will all register different (and yes, I have done that). However, the better ones (as mentioned in the article) are fairly accurate in regard to steps and for the ones that can also calculate it, stairs to. Distance can be accurate if you are willing to accurately calculate your stride length and input it. However, depending on you walking or running speed, stride length changes, so you have to put in a length based on your average walking and/or running pace. Some allow you to enter a value for both.

If the tracker has GPS, then it is much more accurate in relation to distance.

In regard to HR, I have found nothing as accurate as the chest strap versions of HR monitors and find most of the main brands (Garmin, Polar, etc) very accurate. The trackers that do monitor HR, including the Apple Watch, use a pulsating light to detect your capillaries expanding and contracting based on blood volume changes with each heart beat. I have found this to have fairly good accuracy when one is not doing much activity. However, the more active and the more intense the activity and the more variance in the activity (Intervals, sprints, etc), I found them to be fair to poor. I have worn several while also wearing a chest strap monitor and they just do not do as good a job and the difference can be huge.

What I did like about the HR on the activity tracker is they can measure it 24 hours a day so it was good to see what it was during true rest, i.e., like while resting and/or sleeping. It clocked mine at 44-50 bpm during the middle of the night consistently over time, which is about the same when I have checked it against a Chest Strap monitor.

Considering all the above, there is no way they are very accurate in regard to calories as there is too much variance and variables involved.

Currently, I own the Polar HR7 for heart rate monitoring during exercise.

For activity & steps (and fun!), my wife and I both have the Fitbit One, which I have calibrated using both simultaneous GPS and actual mileage and after doing that, find it to have really good accuracy. I tried several of the other trackers including several other FitBit models, and because of the inaccuracy in the HR and the fact I don't like to wear something on my wrist all day, I find the FitBit one to be the best choice. Studies have confirmed its accuracy.

Before pedometers went high tech, my favorite was the Omron 112, which was tested and shown to be very accurate but may no longer be available.

The above is just my opinion based on my experience over the years.

The study below reviews the current data/science on them.

In Health
Jeff

Review
Systematic review of the validity and reliability of consumer-wearable activity trackers
International Journal of Behavioral Nutrition and Physical Activity 2015, 12:159
doi:10.1186/s12966-015-0314-1

http://www.ijbnpa.org/content/12/1/159

Abstract

Background
Consumer-wearable activity trackers are electronic devices used for monitoring fitness- and other health-related metrics. The purpose of this systematic review was to summarize the evidence for validity and reliability of popular consumer-wearable activity trackers (Fitbit and Jawbone) and their ability to estimate steps, distance, physical activity, energy expenditure, and sleep.

Methods
Searches included only full-length English language studies published in PubMed, Embase, SPORTDiscus, and Google Scholar through July 31, 2015. Two people reviewed and abstracted each included study.

Results
In total, 22 studies were included in the review (20 on adults, 2 on youth). For laboratory-based studies using step counting or accelerometer steps, the correlation with tracker-assessed steps was high for both Fitbit and Jawbone (Pearson or intraclass correlation coefficients (CC) > =0.80). Only one study assessed distance for the Fitbit, finding an over-estimate at slower speeds and under-estimate at faster speeds. Two field-based studies compared accelerometry-assessed physical activity to the trackers, with one study finding higher correlation (Spearman CC 0.86, Fitbit) while another study found a wide range in correlation (intraclass CC 0.36–0.70, Fitbit and Jawbone). Using several different comparison measures (indirect and direct calorimetry, accelerometry, self-report), energy expenditure was more often under-estimated by either tracker. Total sleep time and sleep efficiency were over-estimated and wake after sleep onset was under-estimated comparing metrics from polysomnography to either tracker using a normal mode setting. No studies of intradevice reliability were found. Interdevice reliability was reported on seven studies using the Fitbit, but none for the Jawbone. Walking- and running-based Fitbit trials indicated consistently high interdevice reliability for steps (Pearson and intraclass CC 0.76–1.00), distance (intraclass CC 0.90–0.99), and energy expenditure (Pearson and intraclass CC 0.71–0.97). When wearing two Fitbits while sleeping, consistency between the devices was high.

Conclusion
This systematic review indicated higher validity of steps, few studies on distance and physical activity, and lower validity for energy expenditure and sleep. The evidence reviewed indicated high interdevice reliability for steps, distance, energy expenditure, and sleep for certain Fitbit models. As new activity trackers and features are introduced to the market, documentation of the measurement properties can guide their use in research settings.

[Ltldogg]

Thanks Jeff. I purchased the Fitbit One today to check out sleep monitor.

Cheers,
Scott

[JeffN]

Thanks Jeff. I purchased the Fitbit One today to check out sleep monitor.

Cheers,
Scott

We enjoy it and if you calibrate your walking and running speed and enter it, is fairly accurate for steps and distance.

The sleep is more of a novelty as it is judging your sleep time by movement. As a generic monitor it seems to do fairly well though. My wife and I have different sleep patterns and while not perfect, it does get them both close. There is a more sensitive setting you can use, but have not tried it yet.

https://exist.io/blog/fitness-tracker-sleep/

Enjoy!

In Health
Jeff

[JeffN]

As I said about the heart rate function on these trackers

A Class-Action Suit Claims Fitbit Devices Inaccurately Measure Heart Rate

http://m.fastcompany.com/3055198/fast-f ... eart-rate#

Fitbit's wristbands don't give accurate heart rate readings during intense exercise, dissatisfied customers allege.

A group of Fitbit customers filed a class-action suit against the company Tuesday, alleging the Fitbit Charge HR and Surge fitness-tracking wristbands don't accurately measure heart rates during exercise.

"Fitbit marketed these products through aggressive and widespread advertising to consumers who were not only deceived in the devices’ true functionality, but who also were put at a safety risk by trusting the Fitbit Heart Rate Monitors’ inaccurate measurements," Robert Klonoff, one of the customers' lawyers, said in a statement. "Many thousands of consumers paid a premium to get accurate heart rate monitors, and instead got devices that do not work as promised."

According to a complaint filed in California federal court, a cardiologist compared the wristbands' readings to a standard electrocardiogram, and found a "significant degree" of variation.

"At intensities over 110 [beats per minute], the Heart Rate Trackers often failed to record any heart rate at all," according to the complaint. "And even when they did record heart rates, the Heart Rate Trackers were inaccurate by an average of 24.34 bpm, with some readings off by as much as 75 bpm."

In an email to Fast Company, a Fitbit spokeswoman says the company "strongly disagrees with the statements made in the complaint and plans to vigorously defend the lawsuit." Fitbit says its wristbands provide "better overall heart rate tracking than cardio machines at the gym, as it tracks your heart rate continuously even while you’re not at the gym or working out" and aren't intended to be scientific or medical devices.

The company's terms of service require that disputes be resolved through arbitration, but the customers say those contract clauses are hidden in Fitbit's terms of service and not disclosed to customers who buy the wristbands from third-party merchants.

"They are brought to the attention of consumers who purchased at third-party websites and retail locations only after they buy their Fitbits and visit Fitbit’s website to register them," lawyer Jonathan D. Selbin, also representing the Fitbit customers, said in a statement. "Fitbit recently admitted in court documents in an unrelated case that the Fitbit devices cannot function properly without registering them on Fitbit’s website. And just by visiting that website, Fitbit purports to bind you to the arbitration clause and class action ban."

The customers are seeking actual and punitive damages on behalf of customers who bought the devices through third-party merchants and didn't sign on to the arbitration terms.

The same PurePulse-branded heart rate tracking is used in Fitbit's new Blaze smartwatch-like fitness tracker, according to the lawsuit. The Blaze was unveiled earlier this week to a mixed reception and a decline in Fitbit's stock price, as analysts compared it unfavorably to the similar-looking Apple Watch.

[JeffN]

Follow-up to the above lawsuit

Fitbit accuracy questioned in lawsuit
CNN
May 20, 2016

Highlights

- A study finds the pulse technology in Fitbit heartbeat trackers was off 20 beats per minute, on average

- The study was funded by the legal team behind a lawsuit

- Other studies have looked at the accuracy of the devices and found mixed results


http://www.cnn.com/2016/05/20/health/fi ... index.html

In Health
Jeff

[JeffN]

Fitness Trackers Fall Short in Counting Calories

http://everydaydiabetes.com/fitness-tra ... -calories/

(REUTERS) Fitness trackers may be a trendy way to monitor every step we take, but these gadgets are actually pretty bad at keeping tabs on how much energy we burn, a new study suggests.

Scientists pitted 12 devices like the Fitbit Flex and Jawbone Up24 against two proven methods of monitoring energy expenditure – locking people in a room to assess every calorie consumed and burned, or asking people at home to drink specially treated water that makes it possible to detect energy output with a urine test.

“THESE STUDIES DEMONSTRATE THAT EVEN THE MOST POPULAR APPLICATIONS AND DEVICES MAY BE INACCURATE OR HIGHLY VARIABLE.”

In the first experiment, measurements from the fitness trackers deviated from the lab results in a typical day by underestimating energy expenditure by as much as 278 calories or overestimating by up to 204 calories. With the second experiment, the devices ranged from 69 to 590 calories lower than the urine tests.

The results are troubling because when fitness trackers overestimate exercise, people who need more exercise to maintain or lose weight might get too little activity, increasing their risk for obesity and other chronic health problems, said senior study author Motohiko Miyachi of the National Institute of Health and Nutrition in Tokyo, in an email.

Underestimating exercise might be just as dangerous for some people, said Dr. Adam Schoenfeld, a researcher at the University of California, San Francisco and author of an editorial accompanying the study in JAMA Internal Medicine.

“For example, it could be quite dangerous if someone with heart disease had inaccurate recordings of their activity and exercise that was being used to make medical decisions,” Schoenfeld said by email.

“In healthy persons, use of fitness trackers may not be as risky, especially if the information collected is not used for medical decision-making,” Schoenfeld added. “Still, even for healthy users, it may be difficult to promote health and wellness if these devices are proving inaccurate or variable feedback.”

To test the accuracy of fitness trackers for monitoring energy expenditure, Miyachi and colleagues asked nine men and 10 women ages 21 to 50 to wear 12 different devices while participating in the two experiments.

Eight devices used in the experiments are popular with consumers in Japan – Fitbit Flex, Jawbone UP24, Misfit Shine, Epson Pulsense PS-100, Garmin Vivofit, Tanita AM-160, Omron CalorieScan HJA-403C, and Withings Pulse O2.

The other four gadgets have been validated in previous research – Panasonic Actimarker EW 4800, Suzuken Lifecorder EX, Omron Active style Pro HJA-350IT, and ActiGraph GT3X.

For the first experiment, participants went into what’s known as a metabolic chamber, a room specially designed to monitor calories consumed and burned, for 24 hours. They got three meals, and they could work at a desk, exercise on a treadmill, watch television, do housework, and sleep while they were in the room.

In this airtight chamber, scientists can use a technique known as indirect calorimetry to assess energy expenditure by measuring carbon dioxide production and oxygen consumption.

Compared with these measurements, half of the fitness trackers underestimated energy expenditure and the rest overestimated it.

For the second experiment, each participant wore the devices for 15 days and collected urine samples on eight days. Every fitness tracker underestimated energy expenditure, the study found.

It’s possible some of the underestimation might be due to people removing the devices to bathe or to charge batteries, the authors note.

In addition to the small size, other limitations of the study include its reliance on participants who weren’t obese and who didn’t have health problems that would limit their ability to exercise, the authors also note.

Still, the findings suggest that consumers may not have an easy time finding a reliable fitness tracker to monitor exercise, Schoenfeld said.

“It is currently quite challenging to tell which fitness trackers are accurate and reliable and which are not since there aren’t much data available,” Schoenfeld added. “These studies demonstrate that even the most popular applications and devices may be inaccurate or highly variable.”

SOURCE: bit.ly/1RatKFW JAMA Internal Medicine, online March 21, 2016.

Accuracy of Wearable Devices for Estimating Total Energy Expenditure
Comparison With Metabolic Chamber and Doubly Labeled Water Method
JAMA Intern Med. 2016;176(5):702-703. doi:10.1001/jamainternmed.2016.0152.

EXTRACT

This study examines the accuracy of total energy expenditure estimates made by wearable devices compared with measurements made using the metabolic chamber and doubly labeled water methods.

Accurate estimation of energy expenditure is a key element in determining the relationships between aspects of human behavior, physical activity, and overall health.1,2 Although wearable devices for estimating energy expenditure are becoming increasingly popular, there is little evidence regarding their validity.3,4 This study was performed to examine the validity of total energy expenditure estimates made by several wearable devices compared with gold standard measurements for a standardized day (metabolic chamber method) and free-living days (doubly labeled water [DLW] method).

[JeffN]

Original Investigation
September 20, 2016

Effect of Wearable Technology Combined With a Lifestyle Intervention on Long-term Weight Loss
The IDEA Randomized Clinical Trial FREE
JAMA. 2016;316(11):1161-1171. doi:10.1001/jama.2016.12858.

Full Text
http://jama.jamanetwork.com/article.asp ... id=2553448

ABSTRACT
Importance
Effective long-term treatments are needed to address the obesity epidemic. Numerous wearable technologies specific to physical activity and diet are available, but it is unclear if these are effective at improving weight loss.

Objective
To test the hypothesis that, compared with a standard behavioral weight loss intervention (standard intervention), a technology-enhanced weight loss intervention (enhanced intervention) would result in greater weight loss.

Design, Setting, Participants
Randomized clinical trial conducted at the University of Pittsburgh and enrolling 471 adult participants between October 2010 and October 2012, with data collection completed by December 2014.

Interventions
Participants were placed on a low-calorie diet, prescribed increases in physical activity, and had group counseling sessions. At 6 months, the interventions added telephone counseling sessions, text message prompts, and access to study materials on a website. At 6 months, participants randomized to the standard intervention group initiated self-monitoring of diet and physical activity using a website, and those randomized to the enhanced intervention group were provided with a wearable device and accompanying web interface to monitor diet and physical activity.

Main Outcomes and Measures
The primary outcome of weight was measured over 24 months at 6-month intervals, and the primary hypothesis tested the change in weight between 2 groups at 24 months. Secondary outcomes included body composition, fitness, physical activity, and dietary intake.

Results
Among the 471 participants randomized (body mass index [BMI], 25 to <40; age range, 18-35 years; 28.9% nonwhite, 77.2% women), 470 (233 in the standard intervention group, 237 in the enhanced intervention group) initiated the interventions as randomized, and 74.5% completed the study. For the enhanced intervention group, mean baseline weight was 96.3 kg (95% CI, 94.2-98.5) and 24-month weight 89.3 kg (95% CI, 87.1-91.5). For the standard intervention group, mean baseline weight was 95.2 kg (95% CI, 93.0-97.3) and 24-month weight was 92.8 kg (95% CI, 90.6-95.0). Weight change at 24 months differed significantly by intervention group (estimated mean weight loss, 3.5 kg [95% CI, 2.6-4.5} in the enhanced intervention group and 5.9 kg [95% CI, 5.0-6.8] in the standard intervention group; difference, 2.4 kg [95% CI, 1.0-3.7]; P = .002). Both groups had significant improvements in body composition, fitness, physical activity, and diet, with no significant difference between groups.

Conclusions and Relevance
Among young adults with a BMI between 25 and less than 40, the addition of a wearable technology device to a standard behavioral intervention resulted in less weight loss over 24 months. Devices that monitor and provide feedback on physical activity may not offer an advantage over standard behavioral weight loss approaches.

[JeffN]

Perhaps their accuracy is improving though I recently got the new Fitbit Charge HR2 and I still do not find the HR to be very accurate during exercise and the more variation or intensity, the less accurate it is


Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort
J. Pers. Med. 2017, 7(2), 3;

http://www.mdpi.com/2075-4426/7/2/3

Abstract
Background:
The ability to measure activity and physiology through wrist-worn devices
provides an opportunity for cardiovascular medicine. However, the accuracy of commercial devices is largely unknown.

Objective:
To assess the accuracy of seven commercially available wrist-worn devices in estimating heart rate (HR) and energy expenditure (EE) and to propose a wearable sensor evaluation framework.

Methods:
We evaluated the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2. Participants wore devices while being simultaneously assessed with continuous telemetry and indirect calorimetry while sitting, walking, running, and cycling. Sixty volunteers (29 male, 31 female, age 38 ± 11 years) of diverse age, height, weight, skin tone, and tness level were selected. Error in HR and EE was computed for each subject/device/activity combination.

Results:
Devices reported the lowest error for cycling and the highest for walking. Device error was higher for males, greater body mass index, darker skin tone, and walking. Six of the devices achieved a median error for HR below 5% during cycling. No device achieved an error in EE below 20 percent. The Apple Watch achieved the lowest overall error in both HR and EE, while the Samsung Gear S2 reported the highest.

Conclusions:
Most wrist-worn devices adequately measure HR in laboratory-based activities, but poorly estimate EE, suggesting caution in the use of EE measurements as part of health improvement programs. We propose reference standards for the validation of consumer health devices (http://precision.stanford.edu/).


Fitness Bands Fail on Calorie Counts
Scientific American
By Christopher Intagliata
May 24, 2017
https://www.scientificamerican.com/podc ... ie-counts/

[JeffN]

Conclusions and Relevance
Among young adults with a BMI between 25 and less than 40, the addition of a wearable technology device to a standard behavioral intervention resulted in less weight loss over 24 months. Devices that monitor and provide feedback on physical activity may not offer an advantage over standard behavioral weight loss approaches.

I believe this new study explains a possible reason for the above results. It is a reason I hear often from clients.

"But the allure soon faded. After about a month, most of the teenagers had begun to find the monitors chiding and irksome, making them feel lazy if they did not manage 10,000 steps each day. Many also said they now considered themselves more physically inept than they had at the study’s start, often because they were rarely near the top of the activity leader boards."


Study

The Motivational Impact of Wearable Healthy Lifestyle Technologies: A Self-determination Perspective on Fitbits With Adolescents
Charlotte Kerner & Victoria A. Goodyear
American Journal of Health Education Vol. 48 , Iss. 5,2017
http://www.tandfonline.com/doi/full/10. ... ccess=true

Abstract

Background:
Considerable numbers of young people are not meeting physical activity guidelines. Wearable fitness devices can provide opportunities for physical activity promotion.

Purpose:
The aim of the study was to explore whether wearable healthy lifestyle technologies impacted on adolescents’ (13- to 14-year-olds) motivation for physical activity. Methods: The study was a mixed method sequential design. Participants were 84 adolescents (44 girls, 40 boys) from 6 physical education classes. Pupils were issued with a Fitbit to wear for 8 weeks and completed pre-/posttest questionnaires that assessed motivational regulation and psychological need satisfaction. Adolescents also engaged in focus group interviews after wearing the Fitbit for 8 weeks. Quantitative data were analyzed using a repeated measures multivariate analysis of variance (MANOVA) to explore differences between gender and time. Qualitative data analysis was conducted deductively using self-determination theory.

Results:
The quantitative findings identified significant reductions in need satisfaction and autonomous motivation and significant increases in amotivation after 8 weeks. Qualitative evidence suggested short-term increases in motivation through feelings of competition, guilt, and internal pressure.

Discussion:
Findings suggest that healthy lifestyle technology may have negative motivational consequences. Translation to Health Education Practice: Certified Health Education Specialists should support young people to personalize health targets in order to critically engage with normalized health targets.


Mass Media Article

Activity Trackers Don’t Always Work the Way We Want Them To
By GRETCHEN REYNOLDS
NOV. 21, 2017
https://www.nytimes.com/2017/11/21/well ... ation.html

"But a recent study published in The American Journal of Health Education finds that the gadgets frequently have counterproductive impacts on young people’s attitudes about exercise and the capabilities of their own bodies."

[JeffN]

Some more recent thoughts can be found in this thread

https://www.drmcdougallforums.com/viewt ... 22&t=58938


On their ability to accurately measure sleep and sleep details. I love the title

Journal of Sleep Research
17 October 2019 https://doi.org/10.1111/jsr.12926

https://onlinelibrary.wiley.com/doi/ful ... /jsr.12926

The wrist is not the brain: Estimation of sleep by clinical and consumer wearable actigraphy devices is impacted by multiple patient- and device-specific factors

“Sleep specialists should interpret data from these devices with extreme caution and discuss the limitations of these devices with those patients who wish to use or already are using these devices in hopes of improving their sleep. There is an urgent need for collaboration between sleep specialists, engineers and device manufacturers to yield improved sleep tracking devices accurate enough to apply to individual patient decision making, across a variety of sleep and medical disorders.”