4 min read

How To Import All Your Fitbit Data Into R

2019/01/02

Introduction

Fitbit released a new method to download your data. Now instead of only being able to export summary data or use the API, you can download all the relevant JSON files with the click of a button (and a few verification clicks). This post will go through how to download this data and import it into an R data frame.

Download Fitbit Data

Go to the “Data Export” tab under your profile on fitbit.com and by default you will now see a button for exporting data.

image of download button

image of download button

After clicking, you will receive a confirmation email that will allow you to start the download. Once finished, you will have a zip file of all your FitBit data.

Import into R

With this dataset most types of data are separated into many files. Some have a JSON file for each day, others are in month long batches. I’ll start by testing heart rate so I can compare with my previous post How to Download and Analyze your Fitbit Heartrate in R. Instead of getting an API key and calling the server multiple times, we can simply merge our files together.

First, import the relevant libraries.

library(tidyverse)
library(stringr)
library(jsonlite)
library(lubridate)

Get a list of all the data files by specifying your location.

root.data.path <- "../data/MyFitbitData/WillRenius/user-site-export/"
data.files <- list.files(root.data.path)

Now filter out the files for heart rate data.

With read_JSON, there are a couple ways to read in data. One method is to use the default method to read as a list and configure into a data frame.

test.json <- heart.rate.files[1]
heart.rate.list <- read_json(paste0(root.data.path, 
                                    test.json))
[[1]]
[[1]]$dateTime
[1] "12/25/17 04:12:30"

[[1]]$value
[[1]]$value$bpm
[1] 70

[[1]]$value$confidence
[1] 0

Another method is to use the simplifyVector parameter to automatically coerce into a data frame and flatten to flatten the nested value object for each time entry.

test.json <- heart.rate.files[1]
heart.rate.test <- read_json(paste0(root.data.path, 
                                    test.json),
                             simplifyVector = TRUE,
                             flatten = TRUE)
##            dateTime value.bpm value.confidence
## 1 12/25/17 04:12:30        70                0
## 2 12/25/17 04:12:45        70                0
## 3 12/25/17 04:13:00        70                0
## 4 12/25/17 04:13:15        70                0
## 5 12/25/17 04:13:30        70                0
## 6 12/25/17 04:13:45        70                0

The FitBit JSON files are generally formatted very well for these simplify parameters. To make getting each dataset faster we can put all these steps into a function.

# root.data.path <- "../data/MyFitbitData/WillRenius/user-site-export/"

MergeJSONFiles <- function(root.dir, str.match){
  data.files <- list.files(root.dir)
  
  relevant.files <- data.files[str_detect(data.files, str.match)]
  for (file in relevant.files){
    temp.df <- read_json(paste0(root.data.path,file), 
                         simplifyVector = TRUE, 
                         flatten = TRUE)
    if(!exists("merged.df")){
      merged.df <- temp.df
    } 
    else{
      merged.df <- rbind(merged.df, temp.df)
    }
  }
  return(merged.df)
}

Activity

head(MergeJSONFiles(root.data.path, "very_active_minutes"))
##            dateTime value
## 1 12/24/17 00:00:00     0
## 2 12/25/17 00:00:00    10
## 3 12/26/17 00:00:00     0
## 4 12/27/17 00:00:00     0
## 5 12/28/17 00:00:00     2
## 6 12/29/17 00:00:00     7
head(MergeJSONFiles(root.data.path, "moderately_active_minutes"))
##            dateTime value
## 1 12/24/17 00:00:00     0
## 2 12/25/17 00:00:00    35
## 3 12/26/17 00:00:00     0
## 4 12/27/17 00:00:00     0
## 5 12/28/17 00:00:00    20
## 6 12/29/17 00:00:00     7
head(MergeJSONFiles(root.data.path, "lightly_active_minutes"))
##            dateTime value
## 1 12/24/17 00:00:00     5
## 2 12/25/17 00:00:00   143
## 3 12/26/17 00:00:00   139
## 4 12/27/17 00:00:00   162
## 5 12/28/17 00:00:00   151
## 6 12/29/17 00:00:00    79
head(MergeJSONFiles(root.data.path, "sedentary_minutes"))
##            dateTime value
## 1 12/24/17 00:00:00  1435
## 2 12/25/17 00:00:00   784
## 3 12/26/17 00:00:00   748
## 4 12/27/17 00:00:00   782
## 5 12/28/17 00:00:00   769
## 6 12/29/17 00:00:00   909

Calories

head(MergeJSONFiles(root.data.path, "calories"))
##            dateTime value
## 1 12/24/17 00:00:00  1.16
## 2 12/24/17 00:01:00  1.16
## 3 12/24/17 00:02:00  1.16
## 4 12/24/17 00:03:00  1.16
## 5 12/24/17 00:04:00  1.16
## 6 12/24/17 00:05:00  1.16
head(MergeJSONFiles(root.data.path, "distance"))
##            dateTime value
## 1 12/25/17 04:12:00     0
## 2 12/25/17 04:13:00     0
## 3 12/25/17 04:14:00     0
## 4 12/25/17 04:15:00     0
## 5 12/25/17 04:16:00     0
## 6 12/25/17 04:17:00     0

Steps

head(MergeJSONFiles(root.data.path, "steps"))
##            dateTime value
## 1 12/25/17 04:12:00     0
## 2 12/25/17 04:13:00     0
## 3 12/25/17 04:14:00     0
## 4 12/25/17 04:15:00     0
## 5 12/25/17 04:16:00     0
## 6 12/25/17 04:17:00     0

Sleep

Unfortunately, we can’t use our simplification tricks on every data source. Sleep data, in particular, is unfriendly to simplification and flattening. Instead, we’ll have to adjust to only grab the data we’re interested in levels.data and format that as a data frame. Here I still use simplifyVector to return a data frame, but I only grab the $levels$data data frames. This gives the times and stages of sleep during the night.

rm(sleep.df)
data.files <- list.files(root.data.path)

relevant.files <- data.files[str_detect(data.files, "sleep")]

for (file in relevant.files){
  temp.sleep.df <- bind_rows(
    read_json(
      paste0(
        root.data.path,file), 
      simplifyVector = TRUE)$levels$data)
  if(!exists("sleep.df")){
    sleep.df <- temp.sleep.df
  } 
  else{
    sleep.df <- rbind(sleep.df, temp.sleep.df)
  }
}
head(sleep.df)
readRDS("../data/fitbit_export/fitbit_export_sleep_head")
##                  dateTime    level seconds
## 1 2018-01-23T05:45:30.000   asleep     300
## 2 2018-01-23T05:50:30.000 restless      60
## 3 2018-01-23T05:51:30.000    awake      60
## 4 2018-01-23T05:52:30.000   asleep     840
## 5 2018-01-23T06:06:30.000 restless     120
## 6 2018-01-23T06:08:30.000   asleep     900

There are a few other JSON files that you won’t need to merge to read. badges, trophies, and more are available in the export. Have fun exploring it!